Computer-readable storage medium having stored therein information processing program, information processing apparatus, information processing system, and information processing method

ABSTRACT

A pirate ship is displayed on a television being a stationary display device and on a terminal device being a portable display device. When the terminal device is held in a second attitude, the position and the angle of view of a first virtual camera are adjusted, and the pirate ship displayed on the television is zoomed out. When the terminal device is held in a first attitude, the position and the angle of view of the first virtual camera are adjusted and the pirate ship displayed on the television is zoomed in.

CROSS REFERENCE TO RELATED APPLICATION

The disclosure of Japanese Patent Application No. 2011-125648, filed onJun. 3, 2011, is incorporated herein by reference.

FIELD

The technology disclosed herein relates to an information processingprogram, an information processing apparatus, an information processingsystem, and an information processing method which cause images to bedisplayed on a plurality of display devices.

BACKGROUND AND SUMMARY

There have been game systems that cause images to be displayed on aplurality of display devices. For example, in a conventional gamesystem, a handheld game apparatus displays a first game image thatallows users to know positions of first to fourth characters in the gamespace, and a television displays images of limited game space rangesthat correspond to the respective second to fourth characters.

However, in such a conventional game system, although game space imagesare displayed on a plurality of display devices, each player merelyperforms the game by looking at the game space range that corresponds tohis or her own character. Thus, there has been room to improve the gamesystem such that the player is caused to effectively use the pluralityof display devices to play the game.

Therefore, an object of an exemplary embodiment is to provide aninformation processing program, an information processing apparatus, aninformation processing system, and an information processing methodwhich each cause images to be displayed on a plurality of displaydevices, effectively using the plurality of display devices.

In order to solve the above problem, the exemplary embodiment hasemployed the following configurations.

An example of the exemplary embodiment is directed to acomputer-readable storage medium having stored therein an informationprocessing program performed by a computer of an information processingapparatus which causes a stationary display device to display an image.The program causes the computer to perform: setting a first virtualcamera in a virtual space in which a predetermined object is arranged;setting a second virtual camera in a virtual space in which thepredetermined object is arranged; causing the stationary display deviceto display a first image of the virtual space containing thepredetermined object, the first image being taken by the first virtualcamera; causing a portable display device held by a user to display asecond image of the virtual space taken by the second virtual camera;and determining whether the user is viewing the stationary displaydevice. When it is determined that the user is not viewing thestationary display device, the first image is made more difficult to beviewed than that at a time when it is determined that the user isviewing the stationary display device.

The virtual space in which the first virtual camera is set and thevirtual space in which the second virtual camera is set may be the same,or may be different from each other.

According to the above configuration, images of the virtual spacecontaining the predetermined object are displayed on the stationarydisplay device and the portable display device. When it is determinedthat the user is not viewing the stationary display device, it ispossible to make the predetermined object (for example, a pirate ship90A and the like) displayed on the stationary display device moredifficult be viewed than usual. That is, when the user is viewing thestationary display device, it is possible to make the predeterminedobject easier to be viewed, and when the user is not viewing thestationary display device, it is possible to make the predeterminedobject difficult to be viewed. Accordingly, it is possible to controlthe image displayed on the stationary display device. Therefore, it ispossible for the user to perform the game, for example, by alternatelylooking at the stationary display device and the portable displaydevice.

In another configuration, when it is determined that the user is notviewing the stationary display device, the predetermined object in thefirst image may be made difficult to be viewed, by performing at leastone of a process of displaying at least the predetermined object in azoomed-out manner, a process of displaying a predetermined image in apart or a whole of a region containing the predetermined object in thefirst image, a process of blurring at least the predetermined object, aprocess of making at least the predetermined object transparent ortranslucent, and a process of displaying at least the predeterminedobject in a pixelized manner.

The process of displaying a predetermined image may be, for example, aprocess of displaying an image indicating that fog has occurred in apart or the whole of the first image, an image indicating that thescreen has been blacked out, or an image of another object.

According to the above configuration, it is possible to make thepredetermined object difficult to be viewed by various methods.

In another configuration, the program may cause the computer to furtherperform obtaining an attitude of the portable display device. When theattitude of the portable display device is a first attitude, it isdetermined that the user is viewing the stationary display device, andwhen the attitude of the portable display device is a second attitude,it is determined that the user is not viewing the stationary displaydevice.

It should be noted that the attitude of the portable display device maybe obtained by various methods. For example, the attitude of theportable display device may be calculated and obtained based on datafrom a sensor that detects an attitude and that is provided in theportable display device, or the attitude of the portable display devicecalculated in the portable display device may be obtained. Further, forexample, based on images from cameras provided at predeterminedpositions in a space or in the portable display device, the attitude ofthe portable display device may be calculated.

According to the above configuration, based on the attitude of theportable display device, it is possible to make the predetermined objectdisplayed on the stationary display device difficult to be viewed, andto control the display of the stationary display device.

In another configuration, when the attitude of the portable displaydevice is the second attitude, at least the predetermined object may bedisplayed in a zoomed-out manner by changing at least one of a positionand an angle of view of the first virtual camera.

According to the above configuration, when the attitude of the portabledisplay device is the second attitude, it is possible to display thepredetermined object displayed on the stationary display device in azoomed-out manner.

In another configuration, an attitude of the second virtual camera maybe controlled in accordance with the attitude of the portable displaydevice.

According to the above configuration, it is possible to control theattitude of the second virtual camera in accordance with the attitude ofthe portable display device, and to display the image of the virtualspace taken by the second virtual camera on the portable display device.This allows the user to look around the virtual space by changing theattitude of the portable display device.

In another configuration, when the attitude of the portable displaydevice is a predetermined reference attitude, an image taking directionof the second virtual camera may be set to the same image takingdirection as that of the first virtual camera, and the attitude of thesecond virtual camera may be changed in accordance with a change of theattitude of the portable display device from the reference attitude.

According to the above configuration, the image taking direction of thesecond virtual camera can be made the same as that of the image takingdirection of the first virtual camera, by causing the attitude of theportable display device to coincide with the reference attitude. Then,by changing the attitude of the portable display device, it is possibleto cause the portable display device to display images of the virtualspace seen from various directions.

In another configuration, an image taking direction of the first virtualcamera may be a predetermined direction independent of the attitude ofthe portable display device.

According to the above configuration, it is possible to cause thestationary display device to display an image of the virtual space seenfrom a predetermined direction, and to cause the portable display deviceto display an image of the virtual space seen from a direction inaccordance with the attitude of the portable display device.

In another configuration, when it is determined that the user is notviewing the stationary display device, at least the predetermined objectmay be displayed in a zoomed-out manner, by changing at least one of aposition and an angle of view of the first virtual camera.

According to the above configuration, it is possible to display thepredetermined object displayed on the stationary display device in azoomed-out manner, thereby making it difficult to be viewed.

In another configuration, the first image may contain a second objectand the second image may not contain the second object.

According to the above configuration, the image displayed on thestationary display device contains the second object, and the imagedisplayed on the portable display device does not contain the secondobject. Therefore, in order to view the second object, the user needs tolook at the stationary display device, and thus, it is possible to causethe user to look at the stationary display device and the portabledisplay device.

In another configuration, the predetermined object displayed on theportable display device may be more difficult to be viewed than thepredetermined object that is displayed on the stationary display devicewhen it is determined that the user is viewing the stationary displaydevice.

According to the above configuration, the predetermined object difficultto be viewed is displayed on the portable display device. In order toview the predetermined object more easily, the user needs to view thestationary display device, and thus, it is possible to cause the user tolook at the stationary display device and the portable display device.

In another configuration, a position of the second virtual camera may besubstantially the same as a position of the first virtual camera at atime when it is determined that the user is not viewing the stationarydisplay device.

According to the above configuration, when it is determined that theuser is not viewing the stationary display device, the position of thesecond virtual camera can be set to substantially the same position asthat of the first virtual camera

In another configuration, an angle of view of the second virtual cameramay be substantially the same as an angle of view of the first virtualcamera at a time when it is determined that the user is not viewing thestationary display device.

According to the above configuration, when it is determined that theuser is not viewing the stationary display device, the position and theangle of view of the second virtual camera can be set to substantiallythe same as those of the first virtual camera. Accordingly, it ispossible to cause the stationary display device and the portable displaydevice to display images of similar imaging ranges, respectively.

In another configuration, an imaging range of the second virtual cameramay be substantially the same as an imaging range of the first virtualcamera at a time when it is determined that the user is not viewing thestationary display device.

According to the above configuration, when it is determined that theuser is not viewing the stationary display device, the imaging range ofthe second virtual camera can be set to substantially the same as thatof the first virtual camera.

In another configuration, the second attitude may be an attitude inwhich a screen of the portable display device is substantially parallelto the direction of gravity.

According to the above configuration, when the screen of the portabledisplay device is substantially parallel to the direction of gravity, itis possible to determine that the user is not viewing the stationarydisplay device.

In another configuration, the first attitude may be an attitude in whicha screen of the portable display device is substantially perpendicularto the direction of gravity.

According to the above configuration, when the screen of the portabledisplay device is substantially perpendicular to the direction ofgravity, it is possible to determine that the user is viewing thestationary display device.

In another configuration, the computer may further be caused to performadjusting a volume of a sound outputted from the stationary displaydevice, in accordance with a result of the determination of whether theuser is viewing the stationary display device.

According to the above configuration, it is possible to adjust thevolume of the sound outputted from the stationary display device inaccordance with the determination result.

In another configuration, when it is determined that the user is notviewing the stationary display device, the volume of the sound outputtedfrom the stationary display device may be lowered than that at a timewhen it is determined that the user is viewing the stationary displaydevice.

According to the above configuration, when the user is not viewing thestationary display device, it is possible to lower the volume of thesound outputted from the stationary display device.

In another configuration, the portable display device may include atleast one of a gyro sensor and an acceleration sensor. The attitude ofthe portable display device may be calculated based on data outputtedfrom the at least one of the gyro sensor and the acceleration sensor.

According to the above configuration, it is possible to calculate theattitude of the portable display device based on the data from the atleast one of the gyro sensor and the acceleration sensor.

In another configuration, image data indicating the second image may beoutputted to the portable display device. The portable display deviceincludes: an image data obtaining unit that obtains the image dataoutputted from the information processing apparatus; and a display unitthat displays the second image indicated by the image data.

According to the above configuration, the second image is generated inthe information processing apparatus, and the second image is outputtedto the portable display device, whereby the second image can bedisplayed on the portable display device.

In another configuration, the computer may further be caused to performcompressing the image data indicating the second image, and generatingcompressed image data. The compressed image data is outputted to theportable display device. The image data obtaining unit obtains thecompressed image data outputted from the information processingapparatus. The portable display device further includes an imagedecompression unit that decompresses the compressed image data. Thedisplay unit displays the second image decompressed by the imagedecompression unit.

According to the above configuration, the second image is compressed andoutputted from the information processing apparatus to the portabledisplay device. Therefore, even if the image has a large data amount, itis possible to transmit the image from the game apparatus to theportable display device in a short time period.

It should be noted that another example of the exemplary embodiment maybe an information processing apparatus that realizes the above features.In one exemplary embodiment, an information processing system may bestructured by a plurality of components that realize the above featuresand mutually operate with each other. The information processing systemmay be structured by one or a plurality of apparatuses. An exemplaryembodiment may be an information processing method including stepsperformed by the above-described computer.

According to the exemplary embodiment, it is possible to provide aninformation processing program, an information processing apparatus, aninformation processing system, and an information processing methodwhich each cause a plurality of display devices to display images,effectively using the plurality of display devices.

These and other features, aspects and advantages of certain exemplaryembodiments will become more apparent from the following detaileddescription when taken in conjunction with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an external view showing a non-limiting example of a gamesystem 1;

FIG. 2 is a block diagram showing a non-limiting exemplary internalstructure of a game apparatus 3;

FIG. 3 is a perspective view showing a non-limiting exemplary externalstructure of a main controller 8;

FIG. 4 is a perspective view showing a non-limiting exemplary externalstructure of the main controller 8;

FIG. 5 shows a non-limiting exemplary internal structure of the maincontroller 8;

FIG. 6 shows a non-limiting exemplary internal structure of the maincontroller 8;

FIG. 7 is a perspective view showing a non-limiting exemplary externalstructure of a sub-controller 9;

FIG. 8 is a block diagram showing a non-limiting exemplary structure ofa controller 5;

FIG. 9 shows a non-limiting exemplary external structure of a terminaldevice 7;

FIG. 10 shows a non-limiting exemplary state in which a user holds theterminal device 7;

FIG. 11 is a block diagram showing a non-limiting exemplary internalstructure of the terminal device 7;

FIG. 12A shows a non-limiting example of a basic posture of a playerwhen playing the game, in which posture the player holds the terminaldevice 7 in a first attitude;

FIG. 12B shows a non-limiting example of a basic posture of a playerwhen playing the game, in which posture the player holds the terminaldevice 7 in a second attitude;

FIG. 13A shows a non-limiting example of a television game imagedisplayed on a television 2 when the terminal device 7 is held in thefirst attitude;

FIG. 13B shows a non-limiting example of a television game imagedisplayed on the television 2 when the terminal device 7 is held in thesecond attitude;

FIG. 14 shows a non-limiting example of positional relationship betweenobjects arranged in a virtual space of the game;

FIG. 15A shows a non-limiting example of a terminal game image displayedon the LCD 51 of the terminal device 7 at the time when the player isfacing the front of the television 2 while holding the terminal device 7in front of his or her face;

FIG. 15B shows a non-limiting example of a terminal game image displayedon the LCD 51 of the terminal device 7 at the time when the player isfacing to the right relative to the television 2 while holding theterminal device 7 in front of his or her face;

FIG. 16A shows a non-limiting example of a television game imagedisplayed on the television 2 at a first timing after the game of theexemplary embodiment has been executed;

FIG. 16B shows a non-limiting example of a terminal game image displayedon the LCD 51 of the terminal device 7 at a second timing after apredetermined time period has elapsed from the first timing;

FIG. 17 shows a non-limiting example of how the player changes theattitude of the terminal device 7, from a state where the player isholding the terminal device 7 in a lower position (first attitude) to astate where the player is directing the terminal device 7 upwardly;

FIG. 18 shows a non-limiting example of an elapsed time from a firsttiming t1 at which a pirate 92 shot an arrow 94 till a second timing t2at which the arrow 94 reaches the player;

FIG. 19A shows another non-limiting example of a television game imagedisplayed on the television 2 at a first timing t1 after the game of theexemplary embodiment has been executed;

FIG. 19B shows another non-limiting example of a terminal game imagedisplayed on the LCD 51 of the terminal device 7 at a second timing t2after a predetermined time period has elapsed from the first timing t1;

FIG. 20 is a view of a non-limiting example of the player turning to theright relative to the television 2 in response to an instruction fromthe game apparatus 3, viewed from above in the real space.

FIG. 21 shows a non-limiting example of various data used in gameprocessing;

FIG. 22 is a non-limiting example of a main flowchart showing the flowof the game processing performed in the game apparatus 3;

FIG. 23 is a non-limiting example of a flowchart showing in detail theflow of the game control process (step S3) shown in FIG. 22;

FIG. 24 is a side view of a non-limiting example of the terminal device7 rotating a predetermined angle about an X-axis;

FIG. 25 shows a non-limiting example of an image displayed on the LCD 51of the terminal device 7 at a second timing t2 when the terminal device7 is moving; and

FIG. 26 is a non-limiting example of a lock-on frame 99 displayed on theLCD 51 of the terminal device 7.

DETAILED DESCRIPTION OF NON-LIMITING EXAMPLE EMBODIMENTS [1. OverallConfiguration of Game System]

Hereinafter, a game system 1 according to an exemplary embodiment willbe described with reference to the drawings. FIG. 1 is an external viewshowing a non-limiting example of a game system 1. As shown in FIG. 1,the game system 1 includes a stationary display device (hereinafter,referred to as a “television”) 2 typified by, for example, a televisionreceiver, a stationary game apparatus 3, an optical disc 4, a controller5, a marker device 6, and a terminal device 7. In the game system 1, thegame apparatus 3 executes a game process based on a game operation usingthe controller 5, and the television 2 and/or the terminal device 7display a game image obtained in the game process.

Into the game apparatus 3, the optical disc 4 which is an exemplaryinformation storage medium which is exchangeably used for the gameapparatus 3 is detachably inserted. An information processing program(typically, a game program) to be executed by the game apparatus 3 isstored in the optical disc 4. An insertion operating for the opticaldisc 4 is formed on the front surface of the game apparatus 3. The gameapparatus 3 loads and executes the information processing program storedin the optical disc 4 having been inserted through the insertionopening, thereby executing the game process.

The television 2 is connected to the game apparatus 3 through aconnection cord. The television 2 displays a game image obtained in thegame process executed by the game apparatus 3. The television 2 includesa speaker 2 a (FIG. 2), and the speaker 2 a outputs game sound obtainedas a result of the game process. In another exemplary embodiment, thegame apparatus 3 may be integrated with a stationary display device.Further, the game apparatus 3 and the television 2 may wirelesslycommunicate with each other.

The marker device 6 is provided in the vicinity (above a screen inFIG. 1) of a screen of the television 2. As will be described below indetail, a user (a player) is allowed to perform a game operation ofmoving the controller 5, and the marker device 6 is used for causing thegame apparatus 3 to calculate, for example, a movement, a position, andan attitude of the controller 5. The marker device 6 includes twomarkers, that is, a marker 6R and a marker 6L, on both ends thereof.Specifically, the marker 6R (and the marker 6L) is implemented as atleast one infrared light emitting diode (LED), and emits infrared lightforward from the television 2. The marker device 6 is wire-connected (ormay be wirelessly connected) to the game apparatus 3, and the gameapparatus 3 is able to control whether each infrared LED of the markerdevice 6 is to be lit up. The marker device 6 is portable, and a user isallowed to set the marker device 6 at a desired position. In FIG. 1, anexemplary manner is shown in which the marker device 6 is set on thetelevision 2. However, the marker device 6 may be set at any positionand may face in any direction.

The controller 5 provides the game apparatus 3 with operation data basedon an operation performed on the controller 5. In the exemplaryembodiment described herein, the controller 5 includes a main controller8 and a sub-controller 9, and the sub-controller 9 is detachably mountedto the main controller 8. The controller 5 and the game apparatus 3 areable to wirelessly communicate with each other. In the exemplaryembodiment described herein, for example, the Bluetooth (registeredtrademark) technology is used for the wireless communication between thecontroller 5 and the game apparatus 3. In another exemplary embodiment,the controller 5 and the game apparatus 3 may be wire-connected to eachother. Further, although, in FIG. 1, the number of the controllers 5included in the game system 1 is one, the game system 1 may include aplurality of the controllers 5. Namely, the game apparatus 3 cancommunicate with a plurality of controllers, and multiple persons areallowed to play a game by simultaneously using a predetermined number ofcontrollers. A specific structure of the controller 5 will be describedbelow in detail.

The terminal device 7 approximately has such a size as to be held by auser, and the user is allowed to use the terminal device 7 by holdingand moving the terminal device 7 with his/her hand, or positioning theterminal device 7 at any desired position. The terminal device 7includes a liquid crystal display (LCD) 51 operating as display means,and input means (such as a touch panel 52 and a gyro sensor 64 asdescribed below). The structure of the terminal device 7 will bedescribed below in detail. The terminal device 7 and the game apparatus3 can wirelessly communicate with each other (or wired communication maybe used therebetween). The terminal device 7 receivers, from the gameapparatus 3, data of an image (for example, a game image) generated bythe game apparatus 3, and displays the image on the LCD 51. Although, inthe exemplary embodiment described herein, an LCD is used as a displaydevice, the terminal device 7 may have any other display device such asa display device using, for example, electro luminescence (EL). Further,the terminal device 7 transmits, to the game apparatus 3, operation databased on an operation performed on the terminal device 7.

[2. Internal Structure of Game Apparatus 3]

Next, with reference to FIG. 2, a non-limiting exemplary internalstructure of the game apparatus 3 will be described. FIG. 2 is a blockdiagram showing a non-limiting exemplary internal structure of the gameapparatus 3. The game apparatus 3 includes: a central processing unit(CPU) 10; a system LSI 11; an external main memory 12; a ROM/RTC 13; adisk drive 14; an AV-IC 15, and the like.

The CPU 10, serving as a game processor, executes a game program storedin the optical disc 4 to perform a game process. The CPU 10 is connectedto the system LSI 11. In addition to the CPU 10, the external mainmemory 12, the ROM/RTC 13, the disk drive 14, and the AV-IC 15 are alsoconnected to the system LSI 11. The system LSI 11 performs processingsuch as control of data transmission among respective componentsconnected thereto, generation of an image to be displayed, andacquisition of data from an external apparatus. An internalconfiguration of the system LSI 11 will be described below. The externalmain memory 12, which is of a volatile type, stores programs, such as agame program loaded from the optical disc 4 or a flash memory 17, andvarious data. The external main memory 12 is used as a work area and abuffer area for the CPU 10. The ROM/RTC 13 includes a ROM (so-called aboot ROM) storing a program for starting up the game apparatus 3, and aclock circuit (real time clock: RTC) for counting time. The disk drive14 reads, from the optical disc 4, program data, texture data and thelike, and writes the read data into an internal main memory 11 edescribed below, or the external main memory 12.

An input/output processor (PO processor) 11 a, a graphics processor unit(GPU) 11 b, a digital signal processor (DSP) 11 c, a VRAM (video RAM) 11d, and the internal main memory 11 e, are included in the system LSI 11.These components 11 a to 11 e are connected to each other via aninternal bus, which is not shown.

The GPU 11 b, which is a part of rendering means, generates an imageaccording to a graphics command (rendering command) from the CPU 10. TheVRAM 11 d stores data (such as polygon data and texture data) to be usedby GPU 11 b for executing the graphics command. When an image isgenerated, the GPU 11 b generates image data by using the data stored inthe VRAM 11 d. In the exemplary embodiment described herein, the gameapparatus 3 generates both a game image to be displayed by thetelevision 2, and a game image to be displayed by the terminal device 7.Hereinafter, the game image to be displayed by the television 2 may bereferred to as a “television game image”, and the game image to bedisplayed by the terminal device 7 may be referred to as a “terminalgame image”.

The DSP 11 c functions as an audio processor, and generates sound databy using sound data and sound waveform (tone quality) data stored in theinternal main memory 11 e and/or the external main memory 12. In theexemplary embodiment described herein, as game sounds, both a game soundoutputted from the speaker of the television 2, and a game soundoutputted by a speaker of the terminal device 7 are generated, similarlyto the game images. Hereinafter, the game sound outputted by thetelevision 2 may be referred to as a “television game sound”, and thegame sound outputted by the terminal device 7 may be referred to as a“terminal game sound”.

Data of the image and the sound to be outputted by the television 2,among the images and the sounds generated by the game apparatus 3 asdescribed above, is read by the AV-IC 15. The AV-IC 15 outputs the readdata of image to the television 2 via an AV connector 16, and alsooutputs the read data of sound to the speaker 2 a included in thetelevision 2. Thus, the image is displayed by the television 2, and thesound is outputted from the speaker 2 a.

On the other hand, data of the image and the sound to be outputted bythe terminal device 7, among the images and the sounds generated by thegame apparatus 3, is transmitted to the terminal device 7 by theinput/output processor 11 a, and/or the like. The transmission of thedata to the terminal device 7 by the input/output processor 11 a, and/orthe like will be described below.

The input/output processor 11 a executes data reception and transmissionamong the components connected thereto and data downloading from anexternal apparatus. The input/output processor 11 a is connected to theflash memory 17, a network communication module 18, a controllercommunication module 19, an extension connector 20, a memory cardconnector 21, and a codec LSI 27. To the network communication module18, an antenna 22 is connected. To the controller communication module19, an antenna 23 is connected. The codec LSI 27 is connected to aterminal communication module 28, and an antenna 29 is connected to theterminal communication module 28.

The game apparatus 3 is connected to a network such as the Internet, sothat the game apparatus 3 can communicate with an external informationprocessing apparatus (for example, other game apparatuses, variousservers, or various information processing apparatuses). Namely, theinput/output processor 11 a is connected to a network such as theInternet via the network communication module 18 and the antenna 22, tobe able to communicate with the external information processingapparatus connected to the network. The input/output processor 11 aaccesses the flash memory 17 at regular intervals to detect for presenceof data to be transmitted to the network. When the data to betransmitted is detected, the data is transmitted to the network via thenetwork communication module 18 and the antenna 22. Further, theinput/output processor 11 a receives, via the network, the antenna 22and the network communication module 18, data transmitted from theexternal information processing apparatus or data downloaded from adownload server, and stores the received data in the flash memory 17.The CPU 10 executes the game program to read the data stored in theflash memory 17, thereby using the read data on the game program. Theflash memory 17 may store not only the data transmitted and receivedbetween the game apparatus 3 and the external information processingapparatus, but also saved data (result data or intermediate step data ofthe game) of a game played with the game apparatus 3. Further, a gameprogram may be stored in the flash memory 17.

Further, the game apparatus 3 is able to receive the operation datatransmitted from the controller 5. Namely, the input/output processor 11a receives, via the antenna 23 and the controller communication module19, the operation data transmitted from the controller 5, and(temporarily) stores the operation data in a buffer area of the internalmain memory 11 e or the external main memory 12.

Further, the game apparatus 3 is able to transmit to the terminal device7 and receive from the terminal device 7 data of the image, the sound,and the like. When the game image (terminal game image) is transmittedto the terminal device 7, the input/output processor 11 a outputs, tothe codec LSI 27, data of the game image generated in the GPU 11 b. Thecodec LSI 27 subjects, to a predetermined compression process, the imagedata outputted by the input/output processor 11 a. The terminalcommunication module 28 wirelessly communicates with the terminal device7. Therefore, the image data compressed by the codec LSI 27 istransmitted to the terminal device 7 via the antenna 29 by the terminalcommunication module 28. In the exemplary embodiment described herein,the image data transmitted from the game apparatus 3 to the terminaldevice 7 is used for a game. Therefore, if transmission of an image tobe displayed in the game is delayed, operability in the game isadversely affected. Therefore, it is preferable that delay of thetransmission of the image data from the game apparatus 3 to the terminaldevice 7 occurs as little as possible. Therefore, in the exemplaryembodiment described herein, the codec LSI 27 compresses the image databy using a highly efficient compression technique in compliance with,for example, H.264 standard. It is to be noted that other compressiontechniques may be used, or uncompressed image data may be transmittedwhen a communication speed is sufficient. Further, the terminalcommunication module 28 is a communication module approved by, forexample, Wi-Fi, and may perform wireless communication with the terminaldevice 7 at a high speed by using the MIMO (multiple input multipleoutput) techniques adopted in, for example, the IEEE 802.11n standard.Further, another communication mode may be used.

Further, the game apparatus 3 transmits, to the terminal device 7, thesound data as well as the image data. Namely, the input/output processor11 a outputs the sound data generated by the DSP 11 c, through the codecLSI 27, to the terminal communication module 28. The codec LSI 27subjects the sound data to a compression process, similarly to the imagedata. Although the compression mode for the sound data may be any mode,a mode in which the compression rate is high and deterioration of soundis reduced is preferably used. Further, in another exemplary embodiment,sound data, which is not subjected to the compression process, may betransmitted. The terminal communication module 28 transmits thecompressed image data and the compressed sound data, via the antenna 29,to the terminal device 7.

Furthermore, the game apparatus 3 transmits, according to need, variouscontrol data as well as the image data and the sound data describedabove, to the terminal device 7. The control data represents controlinstructions for components included in the terminal device 7, andrepresents, for example, an instruction for controlling lighting of amarker section (a marker section 55 shown in FIG. 10), and aninstruction for controlling imaging of a camera (a camera 56 shown inFIG. 10). The input/output processor 11 a transmits the control data tothe terminal device 7 according to an instruction from the CPU 10.Although the codec LSI 27 does not subject the control data to acompression process in the exemplary embodiment described herein, thecompression process may be performed in another exemplary embodiment.The data transmitted from the game apparatus 3 to the terminal device 7as described above may be encrypted according to need, or may not beencrypted.

Further, the game apparatus 3 is able to receive various data from theterminal device 7. In the exemplary embodiment described herein, theterminal device 7 transmits the operation data, the image data, and thesound data, which will be described below in detail. The datatransmitted from the terminal device 7 is received by the terminalcommunication module 28 via the antenna 29. In the exemplary embodimentdescribed herein, the image data and sound data transmitted from theterminal device 7 are subjected to the compression process which issimilar to that for the image data and sound data transmitted from thegame apparatus 3 to the terminal device 7. Therefore, the received imagedata and sound data are transferred from the terminal communicationmodule 28 to the codec LSI 27, and the codec LSI 27 subjects the imagedata and sound data to a decompression process, and outputs, to theinput/output processor 11 a, the image data and sound data having beensubjected to the decompression process. On the other hand, since theoperation data transmitted from the terminal device 7 has an amount ofdata which is less than an amount of data of an image and a sound, theoperation data may not be subjected to the compression process. Further,encryption may be performed according to need, or may not be performed.Therefore, the operation data is received by the terminal communicationmodule 28, and is thereafter outputted via the codec LSI 27 to theinput/output processor 11 a. The input/output processor 11 a(temporarily) stores the data received from the terminal device 7 in abuffer area of the internal main memory 11 e or the external main memory12.

Further, the game apparatus 3 is able to connect with another deviceand/or an external storage medium. Namely, to the input/output processor11 a, the extension connector 20 and the memory card connector 21 areconnected. The extension connector 20 is a connector, such as a USB oran SCSI, for interface. The extension connector 20 can be connected to amedium such as an external storage medium or a peripheral device such asanother controller, or allows communication with a network by connectingwith a connector for wired communication instead of using the networkcommunication module 18. The memory card connector 21 is a connector forconnecting with an external storage medium such as a memory card. Forexample, the input/output processor 11 a accesses the external storagemedium via the extension connector 20 or the memory card connector 21,to store data in the external storage medium or read data from theexternal storage medium.

The game apparatus 3 has a power button 24, a reset button 25, and anejection button 26. The power button 24 and the reset button 25 areconnected to the system LSI 11. When the power button 24 is pressed soas to be ON, power is supplied to the respective components of the gameapparatus 3 from an external power supply via an AC adapter which is notshown. When the reset button 25 is pressed, the system LSI 11 restarts aboot program for the game apparatus 3. The ejection button 26 isconnected to the disk drive 14. When the ejection button 26 is pressed,the optical disc 4 is ejected from the disk drive 14.

In another exemplary embodiment, some of the components included in thegame apparatus 3 may be implemented as an extension device which isseparated from the game apparatus 3. In this case, for example, theextension device may be connected to the game apparatus 3 via theextension connector 20. Specifically, the extension device includes thecomponents such as the codec LSI 27, the terminal communication module28, and the antenna 29, and the extension device may be detachablyconnected to the extension connector 20. Thus, when the extension deviceis connected to a game apparatus which does not include the componentsdescribed above, the game apparatus can be structured so as to becommunicable with the terminal device 7.

[3. Structure of Controller 5]

Next, with reference to FIG. 3 to FIG. 7, the controller 5 will bedescribed. As described above, the controller 5 includes the maincontroller 8 and the sub-controller 9. FIG. 3 and FIG. 4 are perspectiveviews each showing a non-limiting exemplary external structure of themain controller 8. FIG. 3 is a perspective view showing a non-limitingexample of the main controller 8 as viewed from the top rear sidethereof FIG. 4 is a perspective view showing a non-limiting example ofthe main controller 8 as viewed from the bottom front side thereof

As shown in FIGS. 3 and 4, the main controller 8 includes a housing 31formed by, for example, plastic molding. The housing 31 has asubstantially parallelepiped shape extending in a longitudinal directionfrom front to rear (the Z1 axis direction shown in FIG. 3). The overallsize of the housing 31 is small enough to be held by one hand of anadult or even a child. A user is allowed to perform a game operation bypressing buttons provided on the main controller 8 and moving the maincontroller 8 to change a position and an attitude (tilt) thereof

The housing 31 include a plurality of operation buttons. As shown inFIG. 3, on the top surface of the housing 31, a cross button 32 a, afirst button 32 b, a second button 32 c, an A button 32 d, a minusbutton 32 e, a home button 32 f, a plus button 32 g, and a power button32 h are provided. In the specification described herein, the topsurface of the housing 31 on which the buttons 32 a to 32 h are providedmay be referred to as a “button surface”. On the other hand, on a bottomsurface of the housing 31, a recessed portion is formed, as shown inFIG. 4. On a slope surface on the rear side of the recessed portion, a Bbutton 32 i is provided. The operation buttons 32 a to 32 i are assignedfunctions in accordance with an information processing program executedby the game apparatus 3 according to need. Further, the power button 32h is used for remotely powering the game apparatus 3 body on or off Thehome button 32 f and the power button 32 h each have a top surfacethereof buried in the top surface of the housing 31. Thus, a user isprevented from inadvertently pressing the home button 32 f or the powerbutton 32 h.

On the rear surface of the housing 31, a connector 33 is provided. Theconnector 33 is used for connecting another device (such as thesub-controller 9 or another sensor unit) to the main controller 8.Further, to the right and the left of the connector 33 on the rearsurface of the housing 31, engagement holes 33 a for preventing removalof the other device from being facilitated are provided.

On the rear side of the top surface of the housing 31, a plurality (fourin FIG. 3) of LEDs 34 a to 34 d are provided. The controller 5 (the maincontroller 8) is assigned a controller type (number) so as to bedistinguishable from the other controllers. For example, the LEDs 34 ato 34 d are used for informing a user of the controller type which iscurrently set to controller 5 that he or she is using, or of a remainingbattery power of the controller 5. Specifically, when a game operationis performed by using the controller 5, one of the plurality of LEDs 34a to 34 d is lit up according to the controller type.

Further, the main controller 8 includes an imaging informationcalculation section 35 (FIG. 6), and has, on the front surface of thehousing 31, a light incident surface 35 a of the imaging informationcalculation section 35, as shown in FIG. 4. The light incident surface35 a is formed of a material which allows at least infrared light fromthe markers 6R and 6L to pass therethrough.

A sound hole 31 a for outputting sound to the outside from a speaker 47(FIG. 5) included in the main controller 8 is formed between the firstbutton 32 b and the home button 32 f on the top surface of the housing31.

Next, with reference to FIGS. 5 and 6, an internal structure of the maincontroller 8 will be described. FIG. 5 and FIG. 6 show a non-limitingexemplary internal structure of the main controller 8. FIG. 5 is aperspective view showing a non-limiting exemplary state where an uppercasing (a part of the housing 31) of the main controller 8 is removed.FIG. 6 is a perspective view showing a non-limiting exemplary statewhere a lower casing (a part of the housing 31) of the main controller 8is removed. FIG. 6 is a perspective view showing a non-limitingexemplary reverse side of a substrate 30 shown in FIG. 5.

As shown in FIG. 5, the substrate 30 is fixed inside the housing 31. Ona top main surface of the substrate 30, the operation buttons 32 a to 32h, the LEDs 34 a to 34 d, an acceleration sensor 37, an antenna 45, thespeaker 47, and the like are provided. These components are connected toa microcomputer 42 (see FIG. 6) via lines (not shown) formed on thesubstrate 30 and the like. In the exemplary embodiment described herein,the acceleration sensor 37 is positioned so as to be deviated from thecenter of the main controller 8 in the X1 axis direction. Thus, amovement of the main controller 8 is easily calculated when the maincontroller 8 is rotated about the Z1 axis. Further, the accelerationsensor 37 is positioned in front of the longitudinal (the Z1 axisdirection) center of the main controller 8. Further, the wireless module44 (FIG. 6) and the antenna 45 allow the controller 5 (the maincontroller 8) to act as a wireless controller.

On the other hand, as shown in FIG. 6, at the front edge of the bottommain surface of the substrate 30, the imaging information calculationsection 35 is provided. The imaging information calculation section 35includes an infrared filter 38, a lens 39, an image pickup element 40,and an image processing circuit 41 located in order, respectively, fromthe front of the main controller 8 on the bottom main surface of thesubstrate 30.

Further, on the bottom main surface of the substrate 30, themicrocomputer 42 and the vibrator 46 are provided. The vibrator 46 is,for example, a vibration motor or a solenoid, and is connected to themicrocomputer 42 via the lines formed on the substrate 30 and the like.The main controller 8 is vibrated by an actuation of the vibrator 46according to an instruction from the microcomputer 42. Thus, thevibration is conveyed to a user's hand holding the main controller 8.Thus, a so-called vibration-feedback game is realized. In the exemplaryembodiment described herein, the vibrator 46 is positioned slightly infront of the longitudinal center of the housing 31. Namely, the vibrator46 is positioned near the end portion of the main controller 8 so as tobe deviated from the longitudinal center thereof, and therefore avibration of the entirety of the main controller 8 is enhanced by thevibration of the vibrator 46. Further, the connector 33 is mounted tothe rear edge on the bottom main surface of the substrate 30. Inaddition to the components shown in FIG. 5 and FIG. 6, the maincontroller 8 includes a quartz oscillator for generating a referenceclock for the microcomputer 42, an amplifier for outputting a soundsignal to the speaker 47, and the like.

FIG. 7 is a perspective view showing a non-limiting exemplary externalstructure of the sub-controller 9. The sub-controller 9 includes ahousing 80 formed by, for example, plastic molding. The overall size ofthe housing 80 is small enough to be held by one hand of an adult oreven a child, similarly to the main controller 8. A player is allowed toperform a game operation also with the sub-controller 9 by operatingbuttons and a stick, and changing a position and an attitude of thecontroller itself

As shown in FIG. 7, an analog joystick 81 is provided on the front edgeside (on the Z2-axis positive direction side) of the top surface (on theY2-axis negative direction side) of the housing 80. Further, a frontedge surface is formed on the front edge of the housing 80 so as to beslightly sloped backward, which is not shown. On the front edge surface,a C button and a Z button are provided so as to be aligned in theupward/downward direction (in the Y2-axis direction shown in FIG. 7).The analog joystick 81 and the respective buttons (the C button and theZ button) are assigned functions in accordance with a game programexecuted by the game apparatus 3 according to need. The analog joystick81 and the respective buttons may be collectively referred to as an“operation section 82 (see FIG. 8)”.

The sub-controller 9 includes an acceleration sensor (an accelerationsensor 83 shown in FIG. 8) inside the housing 80, although it is notshown in FIG. 7. In the exemplary embodiment described herein, theacceleration sensor 83 is implemented as the same acceleration sensor asthe acceleration sensor 37 of the main controller 8. However, theacceleration sensor 83 may not be implemented as the same accelerationsensor as the acceleration sensor 37. For example, the accelerationsensor 83 may be an acceleration sensor operable to detect anacceleration for a predetermined one axis or predetermined two axes.

Further, as shown in FIG. 7, one end of a cable is connected to the rearend of the housing 80. The other end of the cable is connected to aconnector (a connector 84 shown in FIG. 8), although it is not shown inFIG. 7. The connector is able to connect with the connector 33 of themain controller 8. Namely, the main controller 8 and the sub-controller9 are connected to each other by connecting between the connector 33 andthe connector 84.

It is to be noted that the shape of each of the main controller 8 andthe sub-controller 9, the shapes of the operation buttons, the number ofthe acceleration sensors and the number of the vibrators, the settingpositions of the acceleration sensors and the vibrators, and the like,which are as described above with reference to FIG. 3 to FIG. 7, aremerely examples. The other shapes, numbers, and setting positions may beused. Further, in the exemplary embodiment described herein, an imagingdirection of the imaging means of the main controller 8 is the Z1 axispositive direction. However, the imaging direction may be any direction.Namely, the position (the light incident surface 35 a of the imaginginformation calculation section 35) of the imaging informationcalculation section 35 of the controller 5 may not be the front surfaceof the housing 31. The imaging information calculation section 35 may beprovided on any other surface on which light from the outside of thehousing 31 can be incident.

FIG. 8 is a block diagram showing a non-limiting exemplary structure ofthe controller 5. As shown in FIG. 8, the main controller 8 includes theoperation section 32 (the operation buttons 32 a to 32 i), the imaginginformation calculation section 35, a communication section 36, theacceleration sensor 37, and the gyro sensor 48. Further, thesub-controller 9 includes the operation section 82 and the accelerationsensor 83. The controller 5 transmits data representing contents of anoperation performed on the controller 5, as operation data, to the gameapparatus 3. In the following description, the operation datatransmitted by the controller 5 may be referred to as “controlleroperation data”, and the operation data transmitted by the terminaldevice 7 may be referred to as “terminal operation data.”

The operation section 32 includes the operation buttons 32 a to 32 idescribed above, and outputs, to the microcomputer 42 of thecommunication section 36, operation button data representing an inputstate (whether or not each of the operation buttons 32 a to 32 i hasbeen pressed) of each of the operation buttons 32 a to 32 i.

The imaging information calculation section 35 is a system for analyzingdata of an image taken by the imaging means, identifying an area thereofhaving a high brightness, and calculating the position of the center ofgravity, the size, and the like of the area. The imaging informationcalculation section 35 has, for example, a maximum sampling period ofabout 200 frames/sec., and therefore can trace and analyze even arelatively fast motion of the controller 5.

The imaging information calculation section 35 includes the infraredfilter 38, the lens 39, the image pickup element 40, and the imageprocessing circuit 41. The infrared filter 38 allows only infrared lightto pass therethrough, among light incident on the front surface of thecontroller 5. The lens 39 collects the infrared light which has passedthrough the infrared filter 38, and outputs the infrared light to theimage pickup element 40. The image pickup element 40 is a solid-stateimage pick-up device such as, for example, a CMOS sensor or a CCDsensor. The image pickup element 40 receives the infrared lightcollected by the lens 39, and outputs an image signal. The markersection 55 of the terminal device 7 and the marker device 6, which areimaging subjects the images of which are taken, are formed of markersfor outputting infrared light. Accordingly, when the infrared filter 38is provided, the image pickup element 40 receives only the infraredlight which has passed through the infrared filter 38, and generatesimage data, so that images of the imaging subjects (the marker section55 and/or the maker device 6) can be accurately taken. Hereinafter, theimage taken by the image pickup element 40 is referred to as a takenimage. The image data generated by the image pickup element 40 isprocessed by the image processing circuit 41. The image processingcircuit 41 calculates a position of the imaging subject in the takenimage. The image processing circuit 41 outputs data of a coordinaterepresenting the calculated position, to the microcomputer 42 of thecommunication section 36. The data representing the coordinate istransmitted as the operation data to the game apparatus 3 by themicrocomputer 42. Hereinafter, the coordinate is referred to as a“marker coordinate”. The marker coordinate represents various values soas to correspond to an attitude (tilt angle) and/or a position of thecontroller 5. Therefore, the game apparatus 3 is able to calculate theattitude and/or the position of the controller 5 by using the markercoordinate.

In another exemplary embodiment, the controller 5 may not include theimage processing circuit 41. The taken image itself may be transmittedfrom the controller 5 to the game apparatus 3. In this case, the gameapparatus 3 includes a circuit or a program having a function equivalentto the function of the image processing circuit 41, thereby calculatingthe marker coordinate.

The acceleration sensor 37 detects an acceleration (including thegravitational acceleration) of the controller 5, that is, a force(including the gravitational force) applied to the controller 5. Theacceleration sensor 37 detects a value of an acceleration (linearacceleration) in the straight line direction along the sensing axisdirection, among accelerations applied to a detection section of theacceleration sensor 37. For example, a multi-axes acceleration sensorhaving two or more axes detects accelerations of components along theaxes, respectively, as an acceleration applied to the detection sectionof the acceleration sensor. It is to be noted that the accelerationsensor 37 is an electrostatic capacitance type MEMS (micro electromechanical system) acceleration sensor. However, another type ofacceleration sensor may be used.

In the exemplary embodiment described herein, the acceleration sensor 37detects linear accelerations in three axial directions, that is, theup/down direction (the Y1 axis direction shown in FIG. 3) of thecontroller 5, the left/right direction (the X1 axis direction shown inFIG. 3) of the controller 5, and the forward/backward direction (the Z1axis direction shown in FIG. 3) of the controller 5. The accelerationsensor 37 detects an acceleration in the straight line direction alongeach axis. Therefore, an output of the acceleration sensor 37 representsa value of the linear acceleration for each of the three axes. Namely,the detected acceleration is represented as a three-dimensional vectorin an X1Y1Z1 coordinate system (a controller coordinate system) definedrelative to the controller 5.

Data (acceleration data) representing an acceleration detected by theacceleration sensor 37 is outputted to the communication section 36. Theacceleration detected by the acceleration sensor 37 is changed so as tocorrespond to an attitude (tilt angle) and a movement of the controller5. Therefore, an attitude and a movement of the controller 5 can becalculated by using the acceleration data obtained by the game apparatus3. In the exemplary embodiment described herein, the game apparatus 3calculates an attitude, a tilt angle, and the like of the controller 5based on the obtained acceleration data.

When a computer such as a processor (for example, the CPU 10) of thegame apparatus 3 or a processor (for example, the microcomputer 42) ofthe controller 5 performs processing based on a signal of anacceleration outputted from the acceleration sensor 37 (and anacceleration sensor 63 described below), additional information relatingto the controller 5 can be inferred or calculated (determined), as oneskilled in the art will readily understand from the description herein.For example, a case where it is anticipated that the computer willperform processing on the assumption that the controller 5 having theacceleration sensor 37 mounted thereto is in a static state (that is, acase where it is anticipated that the computer will perform processingon the assumption that an acceleration detected by the accelerationsensor will include only the gravitational acceleration) will bedescribed. When the controller 5 is actually in the static state, it ispossible to determine whether or not the controller 5 tilts relative tothe gravity direction and to also determine a degree of the tilt, basedon the acceleration having been detected. Specifically, when a statewhere 1G (gravitational acceleration) is applied to a detection axis ofthe acceleration sensor 37 in the vertically downward directionrepresents a reference, it is possible to determine whether or not thecontroller 5 tilts relative to the reference, based on whether 1G(gravitational acceleration) is applied, and to determine a degree oftilt of the controller 5 relative to the reference, based on themagnitude of the detected acceleration. Further, in the case of themulti-axes acceleration sensor 37, when a signal of an acceleration ofeach axis is further subjected to processing, a degree to the tilt ofthe controller 5 relative to the gravity direction can be determinedwith enhanced accuracy. In this case, the processor may calculate a tiltangle of the controller 5 based on an output from the accelerationsensor 37, or may calculate a direction in which the controller 5 tiltswithout calculating the tilt angle. Thus, when the acceleration sensor37 is used in combination with the processor, a tilt angle or anattitude of the controller 5 can be determined

On the other hand, in a case where it is anticipated that the controller5 will be in a dynamic state (a state in which the controller 5 is beingmoved), the acceleration sensor 37 detects an acceleration based on amovement of the controller 5, in addition to the gravitationalacceleration. Therefore, when the gravitational acceleration componentis eliminated from the detected acceleration through a predeterminedprocess, it is possible to determine a direction in which the controller5 moves. Further, when it is anticipated that the controller 5 will bein the dynamic state, an acceleration component based on the movement ofthe acceleration sensor is eliminated from the detected accelerationthrough a predetermined process, whereby it is possible to determine thetilt of the controller 5 relative to the gravity direction. In anotherexemplary embodiment, the acceleration sensor 37 may include an embeddedprocessor or another type of dedicated processor for performing apredetermined process of the acceleration signal detected by embeddedacceleration detection means before the acceleration signal is outputtedto the microcomputer 42. When, for example, the acceleration sensor 37is used for detecting a static acceleration (for example, gravitationalacceleration), the embedded or dedicated processor could convert theacceleration signal to a tilt angle (or another preferable parameter).

The gyro sensor 48 detects angular velocities around three axes (in theexemplary embodiment described herein, the X1, Y1, and Z1 axes). In thedescription herein, a direction of rotation around the X1 axis isreferred to as a pitch direction, a direction of rotation around the Y1axis is referred to as a yaw direction, and a direction of rotationaround the Z1 axis is referred to as a roll direction. The gyro sensor48 may detect angular velocities around the three axes, and the numberof the gyro sensors to be used, and a manner in which the gyro sensorsto be used are combined may be determined as desired. For example, thegyro sensor 48 may be a three-axes gyro sensor, or may be a gyro sensorobtained by combining a two-axes gyro sensor and a one axis gyro sensorwith each other so as to detect angular velocities around the threeaxes. Data representing the angular velocity detected by the gyro sensor48 is outputted to the communication section 36. Further, the gyrosensor 48 may detect an angular velocity around one axis or two axes.

Further, the operation section 82 of the sub-controller 9 includes theanalog joystick 81, the C button, and the Z button as described above.The operation section 82 outputs stick data (referred to as sub-stickdata) representing a direction in which the analog joystick 81 is tiltedand an amount of the tilt of the analog joystick 81, and operationbutton data (referred to as sub-operation button data) representing aninput state (whether or not each button is pressed) of each button, viathe connector 84, to the main controller 8.

Further, the acceleration sensor 83 of the sub-controller 9, which issimilar to the acceleration sensor 37 of the main controller 8, detectsan acceleration (including the gravitational acceleration) of thesub-controller 9, that is, a force (including the gravitational force)applied to the sub-controller 9. The acceleration sensor 83 detectsvalues of accelerations (linear accelerations) in the straight linedirections along predetermined three-axial directions, amongaccelerations applied to a detection section of the acceleration sensor83. Data (referred to as sub-acceleration data) representing thedetected acceleration is outputted via the connector 84 to the maincontroller 8.

As described above, the sub-controller 9 outputs, to the main controller8, sub-controller data including the sub-stick data, the sub-operationbutton data, and the sub-acceleration data described above.

The communication section 36 of the main controller 8 includes themicrocomputer 42, a memory 43, a wireless module 44, and the antenna 45.The microcomputer 42 controls the wireless module 44 for wirelesslytransmitting data obtained by the microcomputer 42 to the game apparatus3, while using the memory 43 as a storage area in order to performprocessing.

The sub-controller data transmitted from the sub-controller 9 isinputted to the microcomputer 42, and temporarily stored in the memory43. Further, data (referred to as main controller data) outputted to themicrocomputer 42 from the operation section 32, the imaging informationcalculation section 35, the acceleration sensor 37, and the gyro sensor48 is temporarily stored in the memory 43. The main controller data andthe sub-controller data are transmitted as the operation data(controller operation data) to the game apparatus 3. Specifically, themicrocomputer 42 outputs, to the wireless module 44, the operation datastored in the memory 43 at a time at which the data is to be transmittedto the controller communication module 19 of the game apparatus 3. Thewireless module 44 uses, for example, the Bluetooth (registeredtrademark) technology to modulate the operation data onto a carrier waveof a predetermined frequency, and emits the low power radio wave signalfrom the antenna 45. Namely, the operation data is modulated into thelow power radio wave signal by the wireless module 44, and transmittedfrom the controller 5. The low power radio wave signal is received bythe controller communication module 19 on the game apparatus 3 side. Thegame apparatus 3 demodulates or decodes the received low power radiowave signal to obtain the operation data. The CPU 10 of the gameapparatus 3 uses the operation data received from the controller 5 toperform a game process. The wireless transmission from the communicationsection 36 to the controller communication module 19 is sequentiallyperformed at predetermined time intervals. Since the game process isgenerally performed at a cycle of 1/60 sec. (as one frame time), datapreferably needs to be transmitted at a cycle of 1/60 sec. or a shortercycle. For example, the communication section 36 of the controller 5outputs the operation data to the controller communication module 19 ofthe game apparatus 3 every 1/200 seconds.

As described above, the main controller 8 is able to transmit the markercoordinate data, the acceleration data, the angular velocity data, andthe operation button data as the operation data representing anoperation performed on the main controller 8. The sub-controller 9 isable to transmit the acceleration data, the sub-stick data, and theoperation button data as the operation data representing an operationperformed on the sub-controller 9. Further, the game apparatus 3executes the game process by using the operation data as a game input.Therefore, by using the controller 5, a user is allowed to perform agame operation of moving the controller 5 itself in addition to aconventional game operation of pressing each operation button. Forexample, a user is allowed to perform, for example, operations oftilting the main controller 8 and/or the sub-controller 9 at desiredattitudes, an operation of indicating a desired position on the screenby using the main controller 8, and operations of moving the maincontroller 8 and/or the sub-controller 9.

Further, although, in the exemplary embodiment described herein, thecontroller 5 does not have display means for displaying a game image,the controller 5 may have display means for displaying, for example, animage indicative of a remaining battery power.

[4. Structure of Terminal Device 7]

Next, a structure of the terminal device 7 will be described withreference to FIGS. 9 to 11. FIG. 9 shows a non-limiting exemplaryexternal structure of the terminal device 7. (a) of FIG. 9 is a frontview showing a non-limiting example of the terminal device 7, (b) ofFIG. 9 is a top view thereof, (c) of FIG. 9 is a right side viewthereof, and (d) of FIG. 9 is a bottom view thereof. FIG. 10 shows anon-limiting exemplary state in which a user holds the terminal device7.

As shown in FIG. 9, the terminal device 7 includes a housing 50 whichapproximately has a horizontally long plate-like rectangular shape. Thehousing 50 is small enough to be held by a user. Therefore, the user isallowed to hold and move the terminal device 7, and change the locationof the terminal device 7.

The terminal device 7 includes an LCD 51 on a front surface of thehousing 50. The LCD 51 is provided near the center of the front surfaceof the housing 50. Therefore, as shown in FIG. 10, by holding thehousing 50 at portions to the right and the left of the LCD 51, a useris allowed to hold and move the terminal device while viewing a screenof the LCD 51. FIG. 10 shows an exemplary case in which a user holds theterminal device 7 horizontally (with the longer sides of the terminaldevice 7 being oriented horizontally) by holding the housing 50 atportions to the right and the left of the LCD 51. However, the user mayhold the terminal device 7 vertically (with the longer sides of theterminal device 7 being oriented vertically).

As shown in (a) of FIG. 9, the terminal device 7 includes, as operationmeans, a touch panel 52 on the screen of the LCD 51. In the exemplaryembodiment described herein, the touch panel 52 is, but is not limitedto, a resistive film type touch panel. However, a touch panel of anytype, such as electrostatic capacitance type touch panel, may be used.The touch panel 52 may be of single touch type or multiple touch type.In the exemplary embodiment described herein, the touch panel 52 has thesame resolution (detection accuracy) as that of the LCD 51. However, theresolution of the touch panel 52 and the resolution of the LCD 51 neednot be the same. Although an input onto the touch panel 52 is usuallyperformed by using a touch pen, a finger of a user, in addition to thetouch pen, may be used for performing an input onto the touch panel 52.The housing 50 may have an opening for accommodating the touch pen usedfor performing an operation on the touch panel 52. Thus, since theterminal device 7 has the touch panel 52, a user is allowed to operatethe touch panel 52 while moving the terminal device 7. That is, the useris allowed to directly (by using the touch panel 52) perform an inputonto the screen of the LCD 51 while moving the screen of the LCD 51.

As shown in FIG. 9, the terminal device 7 has, as operation means, twoanalog sticks 53A and 53B, and a plurality of buttons 54A to 54L. Theanalog sticks 53A and 53B are each a device for designating a direction.The analog sticks 53A and 53B are each configured such that a stick partoperated by a finger of the user is slidable (or tiltable) in anydirection (at any angle in any direction such as the upward, thedownward, the rightward, the leftward, or the diagonal direction)relative to the front surface of the housing 50. The left analog stick53A is provided to the left of the screen of the LCD 51, and the rightanalog stick 53B is provided to the right of the screen of the LCD 51.Therefore, the user is allowed to perform an input for designating adirection by using the analog stick with either the left hand or theright hand. Further, as shown in FIG. 10, the analog sticks 53A and 53Bare positioned so as to be operated by the user holding the right andleft portions of the terminal device 7. Therefore, the user is allowedto easily operate the analog sticks 53A and 53B also when the user holdsand moves the terminal device 7.

The buttons 54A to 54L are each operation means for performing apredetermined input. As described below, the buttons 54A to 54L arepositioned so as to be operated by the user holding the right and leftportions of the terminal device 7 (see FIG. 10). Accordingly, the useris allowed to easily operate the operation means when the user holds andmoves the terminal device 7.

As shown in (a) of FIG. 9, among the operation buttons 54A to 54L, thecross button (direction input button) 54A and the buttons 54B to 54H areprovided on the front surface of the housing 50. Namely, the buttons 54Ato 54H are positioned so as to be operated by a thumb of the user (seeFIG. 10).

The cross button 54A is provided to the left of the LCD 51 below theleft analog stick 53A. That is, the cross button 54A is positioned so asto be operated by the left hand of the user. The cross button 54A iscross-shaped, and is capable of designating an upward, a downward, aleftward, or a rightward direction. The buttons 54B to 54D are providedbelow the LCD 51. The three buttons 54B to 54D are positioned so as tobe operated by the right and left hands of the user. The four buttons54E to 54H are provided to the right of the LCD 51 below the rightanalog stick 53B. Namely the four buttons 54E to 54H are positioned soas to be operated by the right hand of the user. Further, the fourbuttons 54E, 54H, 54F, and 54G are positioned upward, downward,leftward, and rightward, respectively, (with respect to a centerposition of the four buttons). Accordingly, the terminal device 7 maycause the four buttons 54E to 54H to function as buttons which allow theuser to designate an upward, a downward, a leftward, or a rightwarddirection.

As shown in (a), (b), and (c) of FIG. 9, a first L button 541 and afirst R button 54J are provided on diagonally upper portions (an upperleft portion and an upper right portion) of the housing 50.Specifically, the first L button 541 is provided on the left end of theupper side surface of the plate-shaped housing 50 so as to protrude fromthe upper and left side surfaces. The first R button 54J is provided onthe right end of the upper side surface of the housing 50 so as toprotrude from the upper and right side surfaces. In this way, the firstL button 541 is positioned so as to be operated by the index finger ofthe left hand of the user, and the first R button 54J is positioned soas to be operated by the index finger of the right hand of the user (seeFIG. 10).

As shown in (b) and (c) of FIG. 9, leg parts 59A and 59B are provided soas to protrude from a rear surface (i.e., a surface reverse of the frontsurface on which the LCD 51 is provided) of the plate-shaped housing 50,and a second L button 54K and a second R button 54L are provided on theleg parts 59A and 59B, respectively. Specifically, the second L button54K is provided at a slightly upper position on the left side (the leftside as viewed from the front surface side) of the rear surface of thehousing 50, and the second R button 54L is provided at a slightly upperposition on the right side (the right side as viewed from the frontsurface side) of the rear surface of the housing 50. In other words, thesecond L button 54K is provided at a position substantially opposite tothe position of the left analog stick 53A provided on the front surface,and the second R button 54L is provided at a position substantiallyopposite to the position of the right analog stick 53B provided on thefront surface. Thus, the second L button 54K is positioned so as to beoperated by the middle finger of the left hand of the user, and thesecond R button 54L is positioned so as to be operated by the middlefinger of the right hand of the user (see FIG. 10). Further, as shown in(c) of FIG. 9, the leg parts 59A and 59B each have a surface facingdiagonally upward, and the second L button 54K and the second R button54L are provided on the diagonally upward facing surfaces of the legparts 59A and 59B, respectively. Thus, the second L button 54K and thesecond R button 54L each have a button surface facing diagonally upward.Since it is supposed that the middle fingers of the user move verticallywhen the user holds the terminal device 7, the upward facing buttonsurfaces allow the user to easily press the second L button 54K and thesecond R button 54L. Further, the leg parts provided on the rear surfaceof the housing 50 allow the user to easily hold the housing 50.Moreover, the buttons provided on the leg parts allow the user to easilyperform operation while holding the housing 50.

In the terminal device 7 shown in FIG. 9, the second L button 54K andthe second R button 54L are provided on the rear surface of the housing50. Therefore, if the terminal device 7 is placed with the screen of theLCD 51 (the front surface of the housing 50) facing upward, the screenof the LCD 51 may not be perfectly horizontal. Accordingly, in anotherexemplary embodiment, three or more leg parts may be provided on therear surface of the housing 50. In this case, if the terminal device 7is placed on a floor with the screen of the LCD 51 facing upward, thethree or more leg parts contact with the floor (or another horizontalsurface). Thus, the terminal device 7 can be placed with the screen ofthe LCD 51 being horizontal. Such a horizontal placement of the terminaldevice 7 may be achieved by additionally providing detachable leg parts.

The respective buttons 54A to 54L are assigned functions, according toneed, in accordance with a game program. For example, the cross button54A and the buttons 54E to 54H may be used for direction designationoperation, selection operation, and the like, and the buttons 54B to 54Dmay be used for determination operation, cancellation operation, and thelike.

The terminal device 7 includes a power button (not shown) for turningon/off the power of the terminal device 7. The terminal device 7 mayinclude a button for turning on/off screen display of the LCD 51, abutton for performing connection setting (pairing) for connecting withthe game apparatus 3, and a button for adjusting a sound volume ofloudspeakers (loudspeakers 67 shown in FIG. 11).

As shown in (a) of FIG. 9, the terminal device 7 includes a markersection (a marker section 55 shown in FIG. 11) having a marker 55A and amarker 55B, on the front surface of the housing 50. The marker section55 may be provided at any position. In the exemplary embodimentdescribed herein, the marker section 55 is provided above the LCD 51.The markers 55A and 55B are each implemented as one or more infraredLEDs, like the markers 6L and 6R of the marker device 6. The markersection 55 is used, like the marker device 6, for causing the gameapparatus 3 to calculate, for example, a movement of the controller 5(the main controller 8). The game apparatus 3 is capable of controllingthe infrared LEDs of the marker section 55 to be on or off.

The terminal device 7 includes a camera 56 as imaging means. The camera56 includes an image pickup element (e.g., a CCD image sensor or a CMOSimage sensor) having a predetermined resolution, and a lens. As shown inFIG. 9, in the exemplary embodiment describe herein, the camera 56 isprovided on the front surface of the housing 50. Accordingly, the camera56 is capable of taking an image of the face of the user holding theterminal device 7. For example, the camera 56 is capable of taking animage of the user playing a game while viewing the LCD 51. In anotherexemplary embodiment, one or more camera may be included in the terminaldevice 7.

The terminal device 7 has a microphone (a microphone 69 shown in FIG.11) as sound input means. A microphone hole 60 is provided in the frontsurface of the housing 50. The microphone 69 is embedded in the housing50 at a position inside the microphone hole 60. The microphone detectsfor sound, such as user's voice, around the terminal device 7. Inanother exemplary embodiment, one or more microphone may be included inthe terminal device 7.

The terminal device 7 has loudspeakers (loudspeakers 67 shown in FIG.11) as sound output means. As shown in (d) of FIG. 9, loudspeaker holes57 are provided in the lower side surface of the housing 50. Sound isoutputted through the speaker holes 57 from the loudspeakers 67. In theexemplary embodiment described herein, the terminal device 7 has twoloudspeakers, and the speaker holes 57 are provided at positionscorresponding to a left loudspeaker and a right loudspeaker,respectively. The number of loudspeakers included in the terminal device7 may be any number, and additional loudspeakers, in addition to the twoloudspeakers, may be provided in the terminal device 7.

The terminal device 7 includes an extension connector 58 for connectinganother device to the terminal device 7. In the exemplary embodimentdescribed herein, as shown in (d) of FIG. 9, the extension connector 58is provided in the lower side surface of the housing 50. Any device maybe connected to the extension connector 58. For example, a controller (agun-shaped controller or the like) used for a specific game or an inputdevice such as a keyboard may be connected to the extension connector58. If another device need not be connected, the extension connector 58need not be provided.

In the terminal device 7 shown in FIG. 9, the shapes of the operationbuttons and the housing 50, the number of the respective components, andthe positions in which the components are provided, are merely examples.The shapes, numbers, and positions may be different from those describedabove.

Next, an internal structure of the terminal device 7 will be describedwith reference to FIG. 11. FIG. 11 is a block diagram showing anon-limiting exemplary internal structure of the terminal device 7. Asshown in FIG. 11, the terminal device 7 includes, in addition to thecomponents shown in FIG. 9, a touch panel controller 61, a magneticsensor 62, the acceleration sensor 63, the gyro sensor 64, a userinterface controller (UI controller) 65, a codec LSI 66, theloudspeakers 67, a sound IC 68, the microphone 69, a wireless module 70,an antenna 71, an infrared communication module 72, a flash memory 73, apower supply IC 74, a battery 75, and a vibrator 79. These electroniccomponents are mounted on an electronic circuit board and accommodatedin the housing 50.

The UI controller 65 is a circuit for controlling data input to variousinput sections and data output from various output sections. The UIcontroller 65 is connected to the touch panel controller 61, the analogstick 53(the analog sticks 53A and 53B), the operation button 54 (theoperation buttons 54A to 54L), the marker section 55, the magneticsensor 62, the acceleration sensor 63, the gyro sensor 64, and thevibrator 79. Further, the UI controller 65 is connected to the codec LSI66 and the extension connector 58. The power supply IC 74 is connectedto the UI controller 65, so that power is supplied to the respectivecomponents through the UI controller 65. The internal battery 75 isconnected to the power supply IC 74, so that power is supplied from theinternal battery 75. Further, a battery charger 76 or a cable, which issupplied with power from an external power supply, may be connected tothe power supply IC 74 via a connector or the like. In this case, theterminal device 7 can be supplied with power and charged from theexternal power supply by using the battery charger 76 or the cable.Charging of the terminal device 7 may be performed by setting theterminal device 7 on a cradle (not shown) having a charging function.

The touch panel controller 61 is a circuit which is connected to thetouch panel 52, and controls the touch panel 52. The touch panelcontroller 61 generates a predetermined form of touch position data,based on a signal from the touch panel 52, and outputs the touchposition data to the UI controller 65. The touch position datarepresents a coordinate of a position (the position may be a pluralityof positions when the touch panel 52 is a multiple touch type one) atwhich an input is performed on an input surface of the touch panel 52.The touch panel controller 61 reads a signal from the touch panel 52 andgenerates the touch position data every predetermined period of time.Further, various control instructions for the touch panel 52 are outputfrom the UI controller 65 to the touch panel controller 61.

The analog stick 53 outputs, to the UI controller 65, stick datarepresenting a direction in which the stick part operated by a finger ofthe user slides (or tilts), and an amount of the sliding (tilting). Theoperation button 54 outputs, to the UI controller 65, operation buttondata representing an input state of each of the operation buttons 54A to54L (whether or not each of the operation buttons is pressed).

The magnetic sensor 62 detects the magnitude and direction of a magneticfield to detect an orientation. Orientation data representing thedetected orientation is outputted to the UI controller 65. The UIcontroller 65 outputs, to the magnetic sensor 62, a control instructionfor the magnetic sensor 62. Examples of the magnetic sensor 62 include:sensors using, for example, an MI (Magnetic Impedance) device, afluxgate sensor, a hall device, a GMR (Giant Magneto Resistance) device,a TMR (Tunneling Magneto Resistance) device, and an AMR (AnisotropicMagneto Resistance) device. However, any sensor may be adopted as longas the sensor can detect an orientation. Strictly speaking, the obtainedorientation data does not represent an orientation in a place where amagnetic field in addition to the geomagnetism is generated. Even insuch a case, it is possible to calculate a change in the attitude of theterminal device 7 because the orientation data changes when the terminaldevice 7 moves.

The acceleration sensor 63 is provided inside the housing 50. Theacceleration sensor 63 detects the magnitudes of linear accelerationsalong three axial directions (XYZ axial directions shown in (a) of FIG.9), respectively. Specifically, the long side direction of the housing50 is defined as the Z-axial direction, the short side direction of thehousing 50 is defined as the X-axial direction, and the directionorthogonal to the front surface of the housing 50 is defined as theY-axial direction, and the acceleration sensor 63 detects the magnitudesof the linear accelerations in the respective axial directions.Acceleration data representing the detected accelerations is outputtedto the UI controller 65. The UI controller 65 outputs, to theacceleration sensor 63, a control instruction for the accelerationsensor 63. In the exemplary embodiment described herein, theacceleration sensor 63 is, for example, an electrostatic capacitancetype MEMS acceleration sensor. However, in another exemplary embodiment,another type of acceleration sensor may be used. Further, theacceleration sensor 63 may be an acceleration sensor for detecting themagnitude of acceleration in one axial direction or two axialdirections.

The gyro sensor 64 is provided inside the housing 50. The gyro sensor 64detects angular velocities around the three axes of the above-describedX-axis, Y-axis, and Z-axis, respectively. Angular velocity datarepresenting the detected angular velocities is outputted to the UIcontroller 65. The UI controller 65 outputs, to the gyro sensor 64, acontrol instruction for the gyro sensor 64. Any number and anycombination of gyro sensors may be used as long as the angularvelocities around three axes are detected. The gyro sensor 64 mayinclude a two-axes gyro sensor and a one-axis gyro sensor, like the gyrosensor 48. Alternatively, the gyro sensor 64 may be a gyro sensor fordetecting an angular velocity around one axis or two axes.

The vibrator 79 is, for example, a vibration motor or a solenoid. Thevibrator 79 is connected to the UI controller 65. The terminal device 7is vibrated by actuating the vibrator 79 according to an instructionfrom the UI controller 65. The vibration is conveyed to the user's handholding the terminal device 7. Thus, a so-called vibration-feedback gameis realized.

The UI controller 65 outputs, to the codec LSI 66, the operation data(the terminal operation data) including the touch position data, thestick data, the operation button data, the orientation data, theacceleration data, and the angular velocity data, which have beenreceived from the respective components. If another device is connectedto the terminal device 7 through the extension connector 58, datarepresenting operation on the other device may be also included in theoperation data.

The codec LSI 66 is a circuit for subjecting data to be transmitted tothe game apparatus 3 to a compression process, and subjecting datatransmitted from the game apparatus 3 to a decompression process. TheLCD 51, the camera 56, the sound IC 68, the wireless module 70, theflash memory 73, and the infrared communication module 72 are connectedto the codec LSI 66. The codec LSI 66 includes a CPU 77 and an internalmemory 78. Although the terminal device 7 is configured not to perform agame process, the terminal device 7 needs to execute at least a programfor managing the terminal device 7 and a program for communication. Aprogram stored in the flash memory 73 is loaded into the internal memory78 and executed by the CPU 77 when the terminal device 7 is powered on,thereby starting up the terminal device 7. A part of the area of theinternal memory 78 is used as a VRAM for the LCD 51.

The camera 56 takes an image in accordance with an instruction from thegame apparatus 3, and outputs data of the taken image to the codec LSI66. The codec LSI 66 outputs, to the camera 56, a control instructionfor the camera 56, such as an instruction to take an image. The camera56 is also capable of taking a moving picture. That is, the camera 56 iscapable of repeatedly performing image taking, and repeatedly outputtingimage data to the codec LSI 66.

The sound IC 68 is connected to the loudspeakers 67 and the microphone69. The sound IC 68 is a circuit for controlling input of sound datafrom the microphone 69 to the codec LSI 66 and output of sound data tothe loudspeakers 67 from the codec LSI 66. Specifically, when the soundIC 68 receives sound data from the codec LSI 66, the sound IC 68performs D/A conversion on the sound data, and outputs a resultant soundsignal to the loudspeakers 67 to cause the loudspeakers 67 to outputsound. The microphone 69 detects sound (such as user's voice) propagatedto the terminal device 7, and outputs a sound signal representing thesound to the sound IC 68. The sound IC 68 performs A/D conversion on thesound signal from the microphone 69, and outputs a predetermined form ofsound data to the codec LSI 66.

The codec LSI 66 transmits the image data from the camera 56, the sounddata from the microphone 69, and the operation data from the UIcontroller 65, as the terminal operation data, to the game apparatus 3through the wireless module 70. In the exemplary embodiment describedherein, the codec LSI 66 subjects the image data and the sound data to acompression process similar to that performed by the codec LSI 27. Thecompressed image data and sound data, and the terminal operation dataare outputted to the wireless module 70 as transmission data. Theantenna 71 is connected to the wireless module 70, and the wirelessmodule 70 transmits the transmission data to the game apparatus 3through the antenna 71. The wireless module 70 has the same function asthe terminal communication module 28 of the game apparatus 3. That is,the wireless module 70 has a function of connecting to a wireless LAN bya method based on, for example, the IEEE802.11n standard. Thetransmitted data may be encrypted according to need, or may not beencrypted.

As described above, the transmission data transmitted from the terminaldevice 7 to the game apparatus 3 includes the operation data (theterminal operation data), the image data, and the sound data. If anotherdevice is connected to the terminal device 7 through the extensionconnector 58, data received from the other device may be also includedin the transmission data. The infrared communication module 72 performsinfrared communication with another device based on, for example, theIRDA standard. The codec LSI 66 may include, in the transmission data,data received by the infrared communication, and transmit thetransmission data to the game apparatus 3, according to need.

As described above, the compressed image data and sound data aretransmitted from the game apparatus 3 to the terminal device 7. Thesedata are received by the codec LSI 66 through the antenna 71 and thewireless module 70. The codec LSI 66 decompresses the received imagedata and sound data. The decompressed image data is outputted to the LCD51, and an image is displayed on the LCD 51. On the other hand, thedecompressed sound data is outputted to the sound IC 68, and the soundIC 68 outputs sound through the loudspeakers 67.

When control data is included in the data received from the gameapparatus 3, the codec LSI 66 and the UI controller 65 issue controlinstructions for the respective components, according to the controldata. As described above, the control data represents controlinstructions for the respective components (in the exemplary embodimentdescribed herein, the camera 56, the touch panel controller 61, themarker section 55, the sensors 62 to 64, the infrared communicationmodule 72, and the vibrator 79) included in the terminal device 7. Inthe exemplary embodiment describe herein, the control instructionsrepresented by the control data are considered to be instructions tostart and halt (stop) the operations of the above-mentioned components.That is, some components which are not used for a game may be halted toreduce power consumption. In this case, data from the halted componentsare not included in the transmission data transmitted from the terminaldevice 7 to the game apparatus 3. Since the marker section 55 isimplemented as infrared LEDs, the marker section 55 is controlled bysimply turning on/off the supply of power thereto.

As described above, the terminal device 7 includes the operation meanssuch as the touch panel 52, the analog stick 53, and the operationbutton 54. In another exemplary embodiment, however, the terminal device7 may include other operation means instead of or in addition to theseoperation means.

The terminal device 7 includes the magnetic sensor 62, the accelerationsensor 63, and the gyro sensor 64 as sensors for calculating themovement (including the position and the attitude, or a change in theposition and the attitude) of the terminal device 7. In anotherexemplary embodiment, however, the terminal device 7 may include one ortwo of these sensors. In still another exemplary embodiment, theterminal device 7 may include other sensors instead of or in addition tothese sensors.

The terminal device 7 includes the camera 56 and the microphone 69. Inanother exemplary embodiment, however, the terminal device 7 may notinclude the camera 56 and the microphone 69, or may include either ofthe cameral 56 and the microphone 69.

The terminal device 7 includes the marker section 55 as a component forcalculating the positional relation between the terminal device 7 andthe main controller 8 (such as the position and/or the attitude of theterminal device 7 as viewed from the main controller 8). In anotherexemplary embodiment, however, the terminal device 7 may not include themarker section 55. In still another exemplary embodiment, the terminaldevice 7 may include other means as a component for calculating theabove-mentioned positional relation. For example, in another exemplaryembodiment, the main controller 8 may include a marker section, and theterminal device 7 may include an image pickup element. In this case, themarker device 6 may include an image pickup element instead of aninfrared LED.

[5. Outline of Game Processing]

Next, description will be given of the outline of game processingperformed in the game system 1 of the exemplary embodiment. A gamedescribed in the exemplary embodiment is a rhythm game in which a playeroperates the terminal device 7, to the rhythm. The player holds theterminal device 7 and plays the game by changing the attitude of theterminal device 7 while looking at a game image displayed on thetelevision 2 (television game image) and a game image displayed on theterminal device 7 (terminal game image). FIG. 12A shows a basic postureof a player when playing the game, in which posture the player holds theterminal device 7 in a first attitude. FIG. 12B shows a basic posture ofa player when playing the game, in which posture the player holds theterminal device 7 in a second attitude. FIG. 13A shows an example of atelevision game image displayed on a television 2 when the terminaldevice 7 is held in the first attitude. FIG. 13B shows an example of atelevision game image displayed on the television 2 when the terminaldevice 7 is held in the second attitude. FIG. 14 shows positionalrelationship between objects arranged in a virtual space of the game.

As shown in FIG. 12A and FIG. 12B, the player basically faces the screenof the television 2 (hereinafter, “to face the screen of the television2” may be referred to simply as “to face the television 2”), and playsthe game while looking at the television 2 and the LCD 51 of theterminal device 7. Specifically, when the player holds the terminaldevice 7 in the first attitude, the player plays the game, looking at animage displayed on the screen of the television 2 (television gameimage). When the player holds the terminal device 7 in the secondattitude, the player plays the game, looking at an image displayed onthe LCD 51 of the terminal device 7 (terminal game image). As shown inFIG. 12A, the first attitude is an attitude in which the surface onwhich the LCD 51 of the terminal device 7 is provided is directedvertically upward (an attitude in which the Z-axis negative directioncoincides with the direction of gravity) and in which the screen of theLCD 51 of the terminal device 7 is perpendicular to the direction ofgravity. As shown in FIG. 12B, the second attitude is an attitude of theterminal device 7 in which the player holds the terminal device 7 withboth hands, with his or her arms extended straight forward and in whichthe screen of the LCD 51 of the terminal device 7 is parallel to thedirection of gravity.

As shown in FIG. 13A and FIG. 13B, a pirate ship 90A, a pirate 92, a bow93, and an arrow 94 are displayed on the television 2. When the playerholds the terminal device 7 in the first attitude, the pirate 92 isdisplayed in a zoomed-in manner. On the other hand, when the playerholds the terminal device 7 in the second attitude, the pirate 92 isdisplayed in a zoomed-out manner. In this game, as shown in FIG. 14, theviewpoint of the player (the position of a first virtual camera) is setat a ship 91 in the virtual space, the pirate ship 90A is arranged tothe front of the ship 91, a pirate ship 90B is arranged to the right ofthe ship 91, and a pirate ship 90C is arranged to the left of the ship91. That is, in the exemplary embodiment, a game is performed in whichthe ship 91 where the player is on board is surrounded by the pirateship 90A, the pirate ship 90B, and the pirate ship 90C. The image takingdirection of the first virtual camera is fixed, and the gazing point ofthe first virtual camera is fixed to the pirate 92. It should be notedthat the image taking direction of the first virtual camera is notnecessarily fixed, and for example, the image taking direction may bechanged in accordance with an operation performed onto the operationbutton of the terminal device 7 by the user or in accordance with thestate of the game. When the player holds the terminal device 7 in thefirst attitude, the position of the first virtual camera is moved in theimage taking direction of the first virtual camera and a zoom setting ofthe first virtual camera (the angle of view is reduced) is changed,whereby the pirate 92 is displayed in a zoomed-in manner. On the otherhand, when the player changes the attitude of the terminal device 7 tothe second attitude, the position of the first virtual camera is movedin a direction opposite to the image taking direction of the firstvirtual camera and the first virtual camera zooms out, whereby thepirate 92 is displayed in a zoomed-out manner. It should be noted thatin the exemplary embodiment, the pirate ship 90B and the pirate ship 90Care arranged to the right and the left of the ship 91, respectively.However, the pirate ship 90B and the pirate ship 90C may be arranged atany positions. That is, the pirate ship 90B and the pirate ship 90C arearranged at positions at 90 degrees relative to the front of the ship 91in the in the exemplary embodiment. However, these may be arranged atpositions at, for example, 60 degrees relative to the front of the ship91.

FIG. 15A shows an example of a terminal game image displayed on the LCD51 of the terminal device 7 at the time when the player is facing thefront of the television 2 while holding the terminal device 7 in frontof his or her face. FIG. 15B shows an example of a terminal game imagedisplayed on the LCD 51 of the terminal device 7 at the time when theplayer is facing to the right relative to the television 2 while holdingthe terminal device 7 in front of his or her face.

As shown in FIG. 15A, when the player faces the front of the television2, the pirate ship 90A and a part of the ship 91 are displayed on theLCD 51, but the pirate 92, the bow 93, and the arrow 94 are notdisplayed on the LCD 51. A terminal game image is an image that isobtained by a second virtual camera set on the ship 91 taking an imageof the virtual space. The position of the second virtual camera is setto substantially the same as that of the first virtual camera when thepirate 92 is displayed in a zoomed-out manner. The angle of view of thesecond virtual camera is set to substantially the same as that of thefirst virtual camera when the pirate 92 is displayed in a zoomed-outmanner. Therefore, the imaging range of the second virtual camera issubstantially the same as that of the first virtual camera when thepirate 92 is displayed in a zoomed-out manner. When the player faces thefront of the television 2 while holding the terminal device 7 in frontof his or her face as shown in FIG. 12B, the surface opposite to thesurface on which the screen of the LCD 51 of the terminal device 7 isprovided faces the television 2. In such an attitude of the terminaldevice 7 (herein, this attitude may be referred to as “referenceattitude”), the pirate ship 90A which is located to the front of theship 91 is displayed on the LCD 51 of the terminal device 7. The“reference attitude” is an attitude in which the surface opposite to thesurface on which the LCD 51 of the terminal device 7 is provided facesthe television 2 and in which a straight line extended from the terminaldevice 7 in the Z-axis negative direction is substantially perpendicularto the screen of the television 2. When the terminal device 7 is in thereference attitude, the image taking direction of the second virtualcamera coincides with the image taking direction of the first virtualcamera, and the positions of the first virtual camera and the secondvirtual camera substantially coincide with each other.

The first attitude is an attitude in which the screen of the LCD 51 ofthe terminal device 7 is perpendicular to the direction of gravity.Therefore, no matter where the top surface of the terminal device 7 isdirected as long as the screen of the LCD 51 of the terminal device 7 isperpendicular to the direction of gravity, the terminal device 7 isassumed to be in the first attitude. That is, independent of the degreeof rotation of the terminal device 7 about the Z-axis, when the Z-axisnegative direction of the terminal device 7 almost coincides with thedirection of gravity, the terminal device 7 is assumed to be in thefirst attitude. Similarly, the second attitude is an attitude in whichthe screen of the LCD 51 of the terminal device 7 is parallel to thedirection of gravity. Therefore, no matter where the surface opposite tothe surface on which the screen of the LCD 51 of the terminal device 7is provided is directed as long as the screen of the LCD 51 of theterminal device 7 is parallel to the direction of gravity, the terminaldevice 7 is assumed to be in the second attitude. That is, independentof the degree of rotation of the terminal device 7 about the Y-axis,when the Y-axis negative direction of the terminal device 7 almostcoincides with the direction of gravity, the terminal device 7 isassumed to be in the second attitude. In contrast, the referenceattitude is an attitude in which the Y-axis negative direction of theterminal device 7 coincides with the direction of gravity and in whichthe surface opposite to the surface on which the screen of the LCD 51 ofthe terminal device 7 is provided is directed to the television 2. Thatis, the reference attitude is an attitude of the terminal device 7 atthe time when the player stands to the front of the television 2,holding the terminal device 7 with both hands, with his or her armsextended straight forward.

As shown in FIG. 15B, when the player faces to the right relative to thetelevision 2 (see FIG. 20), the pirate ship 90B and a part of the ship91 are displayed on the LCD 51 of the terminal device 7. When the playerfaces to the right relative to the television while holding the terminaldevice 7 in front of his or her face, the left side surface of theterminal device 7 faces the television 2. In such an attitude of theterminal device 7, the pirate ship 90B which is located to the right ofthe ship 91 is displayed on the LCD 51 of the terminal device 7.

As described above, when the player faces to the right relative to thetelevision 2, the second virtual camera also faces to the right, andthus, the pirate ship 90B is displayed on the LCD 51. That is, theattitude of the second virtual camera is changed in accordance with theattitude of the terminal device 7, and the attitude of the secondvirtual camera is determined so as to coincide with the attitude of theterminal device 7. Specifically, the attitude of the second virtualcamera is set in accordance with rotation angles about the respectiveaxes (X, Y, and Z-axes), which are obtained by integrating by timeangular velocities about the respective axes detected by the gyro sensor64. It should be noted that the zoom setting of and the position in theimage taking direction of the second virtual camera are not changed inaccordance with the attitude of the terminal device 7.

On the other hand, even when the player faces to the right relative tothe television 2, the image displayed on the television 2 is notchanged. That is, the attitude of the first virtual camera is notchanged in accordance with the attitude of the terminal device 7.However, as described above, the position and the zoom setting of thefirst virtual camera are changed in accordance with the attitude of theterminal device 7. Specifically, the position of the first virtualcamera is moved from the ship 91 toward the pirate ship 90A inaccordance with the rotation angle of the terminal device 7 about theX-axis, and concurrently the angle of view of the first virtual camerais reduced, whereby the pirate 92 is displayed in a zoomed-in manner asshown in FIG. 13A.

It should be noted that, in the exemplary embodiment, two virtual spacesare defined in the game apparatus 3, and the first virtual camera is setin one virtual space and the second virtual camera is set in the othervirtual space. The same objects (the pirate ship 90A, backgroundobjects, etc.) are arranged in each of the two virtual spaces, and thesame objects are displayed on each screen. Therefore, the television 2and the terminal device 7 display respective images as if they had beenobtained by different virtual cameras imaging the same virtual space.Meanwhile, when FIG. 13B is compared with FIG. 15A, FIG. 13B shows thepirate 92 but FIG. 15A does not show the pirate 92, although they arebasically similar images. The pirate 92 and the like are not arranged inthe other virtual space for which the second virtual camera is provided.Since the terminal device 7 is a portable display device, the dimensionsof the screen are limited, and thus, even if a small image of the pirate92 is displayed on the LCD 51 of the terminal device 7, the playercannot recognize the image. Therefore, the pirate 92 is not displayed onthe LCD 51 of the terminal device 7. It should be noted that, in anotherexemplary embodiment, one virtual space may be defined, and two virtualcameras may be arranged in one virtual space. Alternatively, two virtualspaces may be defined, and identical objects (the pirate ship 90A, etc.)may not be necessarily arranged in each of the virtual spaces. Theobjects may have different appearances as long as they represent thesame object. For example, a first object representing the pirate ship90A is arranged in one virtual space, and a second object representing apirate ship having the same as or a similar appearance to that of thepirate ship 90A may be arranged in the other virtual space. Then, imagesof these objects are taken by the virtual cameras arranged in respectivevirtual spaces, and the taken images may be displayed on the television2 and the terminal device 7, respectively.

Next, the rhythm game of the exemplary embodiment will be described indetail. FIG. 16A shows an example of a television game image displayedon the television 2 at a first timing after the game of the exemplaryembodiment has been executed. FIG. 16B shows an example of a terminalgame image displayed on the LCD 51 of the terminal device 7 at a secondtiming after a predetermined time period has elapsed from the firsttiming. FIG. 17 shows how the player changes the attitude of theterminal device 7, from a state where the player is holding the terminaldevice 7 in a lower position (first attitude) to a state where theplayer is directing the terminal device 7 upwardly.

When the game is started, an image shown in FIG. 13A or FIG. 13B isdisplayed on the television 2 in accordance with the attitude of theterminal device 7, and predetermined music is outputted from the speaker2 a of the television 2. Then, as shown in FIG. 16A, at a predeterminedtiming (first timing), an instruction image 96 indicating that the arrow94 has been shot is displayed on the television 2. The instruction image96 is an image for notifying the player that an arrow has been shot.Moreover, at the first timing, a voice indicating that the arrow 94 hasbeen shot and from which direction the arrow 94 will come (from whichposition the arrow 94 has been shot), and sound effects indicating thatthe arrow 94 has been shot are outputted from the speaker 2 a of thetelevision 2. For example, as shown in FIG. 16A, an instruction image 96showing that the pirate 92 has shot the arrow 94 is displayed at thefirst timing, and concurrently, the pirate 92 issues a voice instruction“Up” to the player (the voice of the pirate 92 is outputted from thespeaker 2 a of the television 2). In response to the instruction image96 and the voice instruction, the player changes the attitude of theterminal device 7 so as to direct the terminal device 7 upward. Itshould be noted that a pirate who shoots the arrow 94, who is differentfrom the pirate 92, may be displayed. Alternatively, when the pirate 92shoots the arrow 94, letter information indicating from which directionthe arrow 94 will come may be displayed on the television 2, in additionto the instruction image 96 and the voice.

As shown in FIG. 17, in order to receive the arrow with the surfaceopposite to the surface on which the screen of the LCD 51 of theterminal device 7 is provided, the player directs that surface upwardlyin the real space. Then, in the case where the attitude of the terminaldevice 7 is being maintained at the second timing after thepredetermined time period has elapsed from the first timing, the imageshown in FIG. 16B is displayed on the LCD 51 of the terminal device 7.Specifically, an image 97 indicating that the arrow 94 has been receivedwith the terminal device 7 and a circular image 98 indicating theposition at which the arrow has been received are displayed on the LCD51 of the terminal device 7. In the exemplary embodiment, conceptually,the terminal device 7 is used as a target for receiving the arrow 94shot by the pirate 92.

As shown in FIG. 16A and FIG. 16B, a scene in which the arrow 94 is shotis displayed on the television 2 at the first timing, and a scene inwhich the arrow 94 has reached the player is displayed on the LCD 51 ofthe terminal device 7 at the second timing. The scene in which the arrow94 is flying toward the player during the time period from the firsttiming to the second timing is not displayed on either the television 2or the LCD 51 of the terminal device 7. Therefore, the player changesthe attitude of the terminal device 7 in time with the sound outputtedfrom the speaker 2 a of the television 2 and the loudspeakers 67 of theterminal device 7. Specifically, the player estimates a timing at whichthe arrow 94 reaches the player based on the music outputted from thespeaker 2 a of the television 2, changes the attitude of the terminaldevice 7 before the arrow 94 reaches the player, and maintains theattitude of the terminal device 7 until that timing. Moreover, theplayer estimates the timing at which the arrow 94 reaches the playerbased on sound effects outputted from the speaker 2 a of the television2 and the loudspeakers 67 of the terminal device 7. The sound effectsare such sounds that would be generated due to the friction between thearrow 94 and air when the arrow 94 flies through the space. The soundeffects are started to be outputted from the television 2 at the firsttiming at which the arrow 94 is shot, and the sound effects outputtedfrom the television 2 become less loud in accordance with the elapsedtime from the first timing. Meanwhile, the sound effects from theterminal device 7 become gradually loud in accordance with the elapsedtime from the first timing, which allows the player to imagine that thearrow 94 is approaching.

FIG. 18 shows an elapsed time from a first timing t1 at which the pirate92 shot the arrow 94 till a second timing t2 at which the arrow 94reaches the player. As described above, concurrently when the game isstarted, predetermined music is started to be outputted from the speaker2 a of the television 2. At a first timing t1 while the music is beingoutputted, the arrow 94 is shot. Then, at a second timing t2 when apredetermined time period has elapsed from the first timing t1 (forexample, three beats), the arrow 94 reaches the player. At the secondtiming t2, in the case where the attitude of the terminal device 7 isthe attitude instructed at the first timing t1, an image 97 indicatingthat the arrow 94 has been received with the terminal device 7 and acircular image 98 are displayed. Further, a sound indicating that thearrow 94 has been received with the terminal device 7 (sound effect) isoutputted from the loudspeakers 67 of the terminal device 7. It shouldbe noted that at the second timing t2, in the case where the attitude ofthe terminal device 7 is not the attitude instructed at the first timingt1, the image 97 indicating that the arrow 94 has been received with theterminal device 7 and the circular image 98 are not displayed. Forexample, a sound, a vibration, or the like outputted from the terminaldevice 7 (vibration of the vibrator 79) notifies the player that theplayer has failed to receive the arrow with the terminal device 7.Further, when another first timing t1 has come after some time elapsed,the pirate 92 issues a next sound instruction.

During a time period from the second timing t2 to the next first timingt1, the player performs a predetermined operation onto the terminaldevice 7 so as to perform an operation of shaking off the arrow 94received with the terminal device 7. When the operation is performed,the arrow 94 is shaken off, and the image 97 and the circular image 98are not displayed on the LCD 51 of the terminal device 7 any more. Thepredetermined operation may be, for example, an operation that generatesan acceleration of a predetermined magnitude in the Z-axis negativedirection of the terminal device 7, or an operation of pressing apredetermined operation button of the terminal device 7.

The first timings t1 and the second timings t2 are stored associatedwith the predetermined music in advance. The first timings and thesecond timings are set, synchronized with the rhythm of thepredetermined music. For example, the first timings t1 are stored in thegame apparatus 3, being associated with the predetermined music inadvance such that they correspond to the timings of the 8th beat, the16th beat, the 24th beat, and the like after the predetermined music isstarted to be reproduced. Second timings t2 are stored in the gameapparatus 3 in advance such that they correspond to the timing of the3rd beat from each first timing t1. Then, at the time when thepredetermined music is started to be reproduced, the first timings andthe second timings are set. The reproduction speed of the predeterminedmusic may be adjusted by the player. When the reproduction speed of thepredetermined music is adjusted, the time period from the start of thereproduction of the music to the first timing and the time period fromthe start of the reproduction of the music to the second timing are alsoadjusted, in accordance with the adjusted reproduction speed. That is,for example, when the predetermined music is reproduced at a fastertempo than usual, the time period from the start of the reproduction ofthe music to the first timing that comes first is shortened, and thetime period from the first timing to the second timing is alsoshortened.

FIG. 19A shows another example of a television game image displayed onthe television 2 at a first timing t1 after the game of the exemplaryembodiment has been executed. FIG. 19B shows another example of aterminal game image displayed on the LCD 51 of the terminal device 7 ata second timing t2 after a predetermined time period has elapsed fromthe first timing t1.

As shown in FIG. 19A, at a first timing t1, a scene in which the pirate92 is indicating right is displayed on the television 2, andconcurrently, the pirate 92 issues a voice instruction “Right” to theplayer (the voice of the pirate 92 is outputted from the speaker 2 a ofthe television 2). In this case, the arrow 94 is not shot from thepirate ship 90A which is located to the front of the ship 91 and thearrow 94 is shot from the pirate ship 90B which is located to the rightof the ship 91. Therefore, FIG. 19A does not show the instruction image96 indicating that the pirate 92 has shot the arrow 94.

Here, in response to the instruction issued by the pirate 92, the playerchanges his or her posture (and the attitude of the terminal device 7)so as to turn to the right relative to the television 2. FIG. 20 is aview of the player turning to the right relative to the television 2 inresponse to an instruction from the game apparatus 3, viewed from abovein the real space. Usually, the player waits for an instruction from thegame apparatus 3, facing to the front of the television 2. At a firsttiming t1, the player receives an instruction from the game apparatus 3,in the form of an action of the pirate 92 indicating right and the voicefrom the speaker 2 a of the television 2. Then, as shown in FIG. 20, theplayer changes his or her posture so as to turn to the right relative tothe television 2 in accordance with the instruction (the whole body isturned to the right by 90 degrees). Then, as shown in FIG. 19B, at asecond timing t2, the image 97 and the circular image 98 indicating thatthe arrow 94 has reached the terminal device 7 are displayed.

Instructions by the pirate 92 as described above are repeatedly issued,to the rhythm of the music. In the game of the exemplary embodiment,based on an instruction by an image displayed on the television 2 and aninstruction by a voice from the television 2, the player operates theterminal device 7 to the rhythm at a right timing. When the player hasperformed the operation in accordance with the instruction, the playercan obtain a high score.

[6. Details of Game Processing]

Next, the game processing performed in the game system will be describedin detail. First, various data used in the game processing will bedescribed. FIG. 21 shows various data used in the game processing.Specifically, FIG. 21 shows main data stored in the main memory (theexternal main memory 12 or the internal main memory 11 e) of the gameapparatus 3. As shown in FIG. 21, a game program 100, terminal operationdata 110, and processing data 120 are stored in the main memory of thegame apparatus 3. It should be noted that data used to play the game,such as image data of various objects that appear in the game and sounddata used in the game, are stored in the main memory, in addition to thedata shown in FIG. 21.

A part or the whole of the game program 100 is read from the opticaldisc 4, at an appropriate timing after the game apparatus 3 is poweredon, and is stored in the main memory. It should be noted that, the gameprogram 100 may be obtained from the flash memory 17 or an externaldevice of the game apparatus 3 (for example, via the Internet), insteadof the optical disc 4. A part of the game program 100 (for example, aprogram for calculating an attitude of the terminal device 7) may bestored in the game apparatus 3 in advance.

The terminal operation data 110 is data indicating an operationperformed by the player onto the terminal device 7, and is outputted(transmitted) from the terminal device 7 based on the operationperformed onto the terminal device 7. The terminal operation data 110 istransmitted from the terminal device 7, obtained by the game apparatus3, and stored in the main memory. The terminal operation data 110includes angular velocity data 111, acceleration data 112, orientationdata 113, and operation button data 114. In addition to these types ofdata, the terminal operation data 110 further includes data indicatingthe position at which the touch panel 52 of the terminal device 7 istouched (touch position), and the like. When the game apparatus 3obtains terminal operation data from a plurality of terminal devices 7,the game apparatus 3 may cause the terminal operation data 110transmitted from each terminal device 7 to be stored separately in themain memory.

The angular velocity data 111 is data indicating an angular velocitydetected by the gyro sensor 64 in the terminal device. Here, the angularvelocity data 111 indicates angular velocities about the respective axesof the fixed XYZ coordinate system in the terminal device 7 (see FIG.9). However, in another exemplary embodiment, the angular velocity data111 may indicate one or more angular velocities about any one or moreaxes, respectively.

The acceleration data 112 is data indicating an acceleration detected bythe acceleration sensor 63 of the terminal device 7. Here, theacceleration data 112 indicates accelerations about the respective axesof the fixed XYZ coordinate system in the terminal device 7 (see FIG.9).

The orientation data 113 is data indicating an orientation detected bythe magnetic sensor 62 of the terminal device 7.

The operation button data 114 is data indicating whether the operationbuttons 54A to 54L provided in the terminal device 7 are pressed.

The processing data 120 is data used in the game processing (FIG. 22)described below. The processing data 120 includes attitude data 121,first virtual camera data 122, second virtual camera data 123, andstatus data 124. In addition to the data shown in FIG. 21, theprocessing data 120 further includes various data used in the gameprocessing, such as data regarding scores.

The attitude data 121 is data indicating an attitude of the terminaldevice 7. The attitude of the terminal device 7 is expressed, forexample, in a rotation matrix representing a rotation from the referenceattitude to the current attitude. The attitude of the terminal device 7may be expressed by three angles (rotation angles about the respectiveXYZ axes). The attitude data 121 is calculated based on the angularvelocity data 111 included in the terminal operation data 110 from theterminal device 7. Specifically, the attitude data 121 is calculated byintegrating by time angular velocities about the X-axis, the Y-axis, andthe Z-axis, respectively, detected by the gyro sensor 64. It should benoted that the attitude of the terminal device 7 may not be necessarilycalculated based on the angular velocity data 111 indicating the angularvelocity detected by the gyro sensor 64, and may be calculated based onthe acceleration data 112 indicating the acceleration detected by theacceleration sensor 63 and the orientation data 113 indicating theorientation detected by the magnetic sensor 62. Further, the attitudemay be calculated by correcting, based on the acceleration data and theorientation data, an attitude which has been calculated based on theangular velocity. Moreover, pieces of attitude data 121 indicatingattitudes of the terminal device 7 in a predetermined number of pastframes are chronologically stored in the main memory.

The first virtual camera data 122 includes data indicating a positionand an attitude of the first virtual camera in the virtual space wherethe first virtual camera is set, and data indicating a zoom setting ofthe first virtual camera (setting of the angle of view). As describedabove, the first virtual camera is a virtual camera for generating atelevision game image. Although the image taking direction (attitude) ofthe first virtual camera is fixed, the position and the zoom setting ofthe first virtual camera are changed in accordance with the attitude ofthe terminal device 7.

The second virtual camera data 123 is data indicating a position and anattitude of the second virtual camera in the virtual space where thesecond virtual camera is set. As described above, the second virtualcamera is a virtual camera for generating a terminal game image.Although the position of the second virtual camera is fixed, theattitude of the second virtual camera is changed in accordance with theattitude of the terminal device 7.

The status data 124 is data indicating which of the first attitude andthe second attitude the attitude of the terminal device 7 is.

Next, the game processing performed in the game apparatus 3 will bedescribed in detail with reference to FIG. 22 and FIG. 23. FIG. 22 is amain flowchart showing the flow of the game processing performed in thegame apparatus 3. When the game apparatus 3 is powered on, the CPU 10 ofthe game apparatus 3 executes the boot program stored in the boot ROMnot shown, to initialize units such as the main memory. Then, the gameprogram stored in the optical disc 4 is loaded onto the main memory andthe game program is started to be performed by the CPU 10. The flowchartshown in FIG. 22 shows processing performed after the above process iscompleted. It should be noted that, the game apparatus 3 may beconfigured such that the game program is executed immediately after thepower is turned on. Alternatively, the following configuration may beemployed: a built-in program that causes a predetermined menu screen tobe displayed is firstly executed immediately after the power is turnedon, and then, for example, the user performs a selection operation onthe menu screen to issue an instruction of starting the game, wherebythe game program is executed.

It should be noted that the processes of the steps in the flowcharts ofFIG. 22 and FIG. 23 are merely an example, and the sequence of theprocesses of the steps may be changed as long as the same result isobtained. Also, the values and the like of variables and constants aremerely an example, and other values may be employed as appropriate. Inthe exemplary embodiment, description will be given under an assumptionthat all the processes of the steps in the flowcharts are performed bythe CPU 10. However, processes of some of the steps in the flowchartsmay be performed by a processor or a dedicated circuit other than theCPU 10.

First, in step S1, the CPU 10 performs an initial process. The initialprocess is a process of constructing virtual spaces, arranging objects(the ship 91, the pirate ship 90A, the pirate 92, the arrow 94, thefirst virtual camera, the second virtual camera, etc.) that will appearin the virtual spaces at initial positions, and setting initial valuesof various parameters used in the game processing.

Moreover, in step S1, an initial process for the terminal device 7 isperformed. For example, an image for causing the player to hold theterminal device 7 in the reference attitude and to press a predeterminedoperation button of the terminal device 7 while maintaining the attitudeis displayed on the television 2. As a result of the initial process ofthe terminal device 7, rotation angles about the respective XYZ axes inthe terminal device 7 are set to 0. It should be noted that, in theinitial process, only the rotation angle about the Z-axis (rotationangle about the axis perpendicular to the LCD 51 of the terminal device7) may be set to 0 and rotation angles about the X-axis and the Y-axismay be set based on the acceleration detected by the acceleration sensor63. The game apparatus 3 can calculate how much the terminal device 7 istilted relative to the direction of gravity, based on the direction ofgravity detected by the acceleration sensor 63 of the terminal device 7.However, the game apparatus 3 cannot know which direction the terminaldevice 7 is directed (how much the terminal device 7 is rotated aboutthe vertically downward axis direction) only based on the direction ofgravity detected by the acceleration sensor 63. Therefore, in step S1,the player is caused to hold the terminal device 7 such that apredetermined surface of the terminal device 7 (for example, the surfaceopposite to the surface on which the LCD 51 is provided) faces thetelevision 2, and the rotation angle of the terminal device 7 about theZ-axis is set to 0, thereby initializing the attitude of the terminaldevice 7. Accordingly, using the attitude of the terminal device 7 atthe time of the initialization as a reference attitude, the gameapparatus 3 can calculate a change of the attitude of the terminaldevice 7 from the reference attitude, based on the angular velocitydetected by the gyro sensor 64.

When a predetermined operation button of the terminal device 7 ispressed and the initial process of the terminal device 7 is completed,the CPU 10 starts reproduction of predetermined music, and then performsthe process of step S2. Thereafter, a processing loop composed of theprocesses of step S2 to S8 is repeatedly performed once in apredetermined time period (one frame time; for example, 1/60 second).

In step S2, the CPU 10 obtains the terminal operation data 110 which hasbeen transmitted from the terminal device 7 and stored in the mainmemory. The terminal device 7 repeatedly transmits the terminaloperation data 110 to the game apparatus 3. In the game apparatus 3, theterminal communication module 28 sequentially receives terminaloperation data, and the I/O processor 11 a sequentially stores thereceived terminal operation data in the main memory. It is preferablethat the interval between a transmission and a reception performedbetween the terminal device 7 and the game apparatus 3 is shorter thanthe game processing period, and for example, it is 1/200 sec. In stepS2, the CPU 10 reads the latest terminal operation data 110 from themain memory. After step S2, the process of step S3 is performed.

In step S3, the CPU 10 performs a game control process. The game controlprocess is a process of advancing the game in accordance with a gameoperation performed by the player. Hereinafter, with reference to FIG.23, the game control process will be described in detail.

FIG. 23 is a flowchart showing in detail the flow of the game controlprocess (step S3) shown in FIG. 22.

In step S11, the CPU 10 calculates an attitude of the terminal device 7based on the angular velocity data 111. Specifically, the CPU 10calculates the attitude of the terminal device 7, based on the angularvelocity data 111 obtained in step S2 and the attitude data 121 storedin the main memory. More specifically, the CPU 10 calculates rotationangles about the respective axes (X-axis, Y-axis, and Z-axis) obtainedby multiplying, by one frame time, angular velocities about therespective axes indicated by the angular velocity data 111 obtained instep S2. The rotation angles about the respective axes calculated inthis manner are rotation angles of the terminal device 7 about therespective axes (rotation angles in one frame time) during a time periodfrom the time when an immediately preceding processing loop wasperformed to the time when the current processing loop is performed.Next, the CPU 10 adds the calculated rotation angles about therespective axes (rotation angles in one frame time) to the rotationangles of the terminal device 7 about the respective axes indicated bythe attitude data 121, and thereby calculates the latest rotation anglesof the terminal device 7 about the respective axes (the latest attitudeof the terminal device 7). Further, the calculated attitude may becorrected based on the acceleration. Specifically, when the amount ofmotion of the terminal device 7 is little, the direction of theacceleration detected by the acceleration sensor 63 can be considered asthe direction of gravity. Therefore, when the amount of motion of theterminal device 7 is little, the attitude may be corrected such that thedirection of gravity calculated based on the attitude calculated basedon the angular velocity approximates to the direction of theacceleration detected by the acceleration sensor 63. Moreover, thecalculated attitude may further be corrected based on the orientationdata 113. Specifically, how much the terminal device 7 is rotated aboutthe axis in the vertically downward direction can be determined based onthe orientation detected by the magnetic sensor 62 at the time when theinitial process was performed in step 51 and based on the orientationcurrently detected by the magnetic sensor 62. Thus, the attitude of theterminal device 7 may be corrected based on the orientation data 113.Then, the CPU 10 stores the calculated, latest attitude of the terminaldevice 7 as the attitude data 121, in the main memory. The latestattitude of the terminal device 7 calculated in this manner indicates,using the attitude at the time of the initialization process (the timewhen the initialization was performed in step S1) as a referenceattitude, rotation angles of the terminal device 7 about the respectiveaxes from the reference attitude. Specifically, the attitude data 121indicating the attitude of the terminal device 7 is data representing arotation matrix. After the process in step S11, the CPU 10 performs theprocess of step S12.

In step S12, the CPU 10 sets a position and an angle of view of thefirst virtual camera in accordance with the attitude of the terminaldevice 7. Specifically, the CPU 10 determines whether the attitude ofthe terminal device 7 calculated in step S11 is the first attitude orthe second attitude, and performs setting of the first virtual camerabased on the determination result. More specifically, the CPU 10 firstdetermines whether the rotation angle of the terminal device 7 about theX-axis is greater than or equal to a predetermined threshold value (forexample, 45 degrees).

FIG. 24 is a side view of the terminal device 7 rotating a predeterminedangle about an X-axis. As shown in FIG. 24, when the rotation angle ofthe terminal device 7 about the X-axis is greater than or equal to thepredetermined threshold value, the CPU 10 determines that the attitudeof the terminal device 7 is the second attitude. On the other hand, whenthe rotation angle of the terminal device 7 about the X-axis is lessthan the predetermined threshold value, the CPU 10 determines that theattitude of the terminal device 7 is the first attitude. The CPU 10stores the determination result in the main memory, as the status data124. It should be noted that the CPU 10 changes the predeterminedthreshold value depending on whether the current attitude of theterminal device 7 is the first attitude or the second attitude. Forexample, when the current attitude is the first attitude, the CPU 10 mayuse 45 degrees as the predetermined threshold value, and when thecurrent attitude is the second attitude, the CPU 10 may use 30 degreesas the predetermined threshold value.

Next, the CPU 10 sets the position and the angle of view of the firstvirtual camera. Specifically, when the attitude of the terminal device 7is the first attitude, the CPU 10 sets the position of the first virtualcamera to a predetermined position (a position nearer to the pirate ship90A) between the position of the ship 91 and the position of the pirateship 90A, and sets the angle of view of the first virtual camera to aminimum value. Accordingly, when an image of the virtual space is takenby the first virtual camera, the pirate ship 90A (the pirate 92) iszoomed in and thus displayed in a zoomed-in manner. On the other hand,when the attitude of the terminal device 7 is the second attitude, theCPU 10 sets the position of the first virtual camera to the position ofthe ship 91 and sets the angle of view of the first virtual camera to amaximum value. Accordingly, when an image of the virtual space is takenby the first virtual camera, the pirate ship 90A is zoomed out, andthus, is displayed in a zoomed-out manner. It should be noted that theCPU 10 moves the first virtual camera and changes the angle of view ofthe first virtual camera over a predetermined time period. Therefore, ascene in which the pirate 92 is gradually zoomed in or zoomed out isdisplayed on the television 2. After the process of step S12 iscompleted, the CPU 10 performs the process of step S13 next.

In step S13, the CPU 10 sets the volume of the sound outputted from thetelevision 2, in accordance with the attitude of the terminal device 7.Specifically, when the attitude of the terminal device 7 calculated instep S11 is the second attitude, the CPU 10 lowers the volume of thesound (the voice of the pirate 92 and the music) outputted form thetelevision 2, compared with that in the case of the first attitude.Then, the CPU 10 performs the process of step S14.

In step S14, the CPU 10 sets the attitude of the second virtual camerain accordance with the attitude of the terminal device 7. Specifically,the CPU 10 sets the attitude of the second virtual camera so as tocoincide with the attitude of the terminal device 7 calculated in stepS11, and stores it as the second virtual camera data 123 in the mainmemory. Accordingly, for example, when the surface opposite to thesurface on which the LCD 51 of the terminal device 7 is provided facesthe television 2 (when the player faces the television 2), the attitudeof the second virtual camera is set such that the second virtual camerafaces the pirate ship 90A. On the other hand, for example, when thesurface on which the LCD 51 of the terminal device 7 is provided facesthe television 2 (when the player faces opposite to the television 2),the attitude of the second virtual camera is set such that the secondvirtual camera faces opposite to the pirate ship 90A. Then, the CPU 10performs the process of step S15.

In step S15, the CPU 10 determines whether the time is a first timingt1. Specifically, the CPU 10 determines whether the time is a firsttiming t1, based on an elapsed time from the start of the reproductionof the predetermined music in step S1. When the determination result isaffirmative, the CPU 10 performs the process of step S16 next. On theother hand, when the determination result is negative, the CPU 10performs the process of step S18 next.

In step S16, the CPU 10 starts a process of shooting the arrow 94.Specifically, the CPU 10 determines, based on a predetermined algorithm,from which direction to shoot the arrow 94 (front, up, right, left,etc.), and starts the process for shooting the arrow 94 from thedetermined direction. Accordingly, when an image of the pirate 92 istaken by the first virtual camera, a scene in which the pirate 92 shootsthe arrow 94 (FIG. 16A) is displayed on the television 2, and a scene inwhich the pirate 92 indicates the direction from which the arrow 94 willcome (FIG. 19A) is displayed on the television 2. Moreover, as a processof shooting the arrow, the CPU 10 reproduces a voice indicating fromwhich direction the arrow 94 will come and sound effects indicating thatthe arrow 94 has been shot. A plurality of pieces of sound data arestored in the main memory, and the CPU 10 selects a piece of sound datacorresponding to the determined direction. Thus, a voice indicating fromwhich direction the arrow 94 will come (for example, “Right”) isoutputted from the speaker 2 a of the television 2. Then, the CPU 10performs the process of step S17.

In step S17, the CPU 10 starts measuring an elapsed time from the firsttiming t1. After performing the process of step S17, the CPU 10 ends thegame control process shown in FIG. 23.

Meanwhile, in step S18, the CPU 10 determines whether the arrow 94 hascurrently been shot. The process here, as shown in FIG. 18, is a processof determining whether the current time is in a time period from thefirst timing t1 to the second timing t2. Specifically, based on theelapsed time from the time when the measurement was started in step S17,the CPU 10 determines whether the arrow 94 has currently been shot. Whenthe determination result is affirmative, the CPU 10 performs the processof step S19 next. On the other hand, when the determination result isnegative, the CPU 10 performs the process of step S22 next.

In step S19, the CPU 10 determines whether the current time is a secondtiming t2. Specifically, based on an elapsed time from the time when themeasurement was started in step S17, the CPU 10 determines whether thecurrent time is a second timing t2. When the determination result isaffirmative, the CPU 10 performs the process of step S20 next. On theother hand, when the determination result is negative, the CPU 10performs the process of step S21 next.

In step S20, the CPU 10 performs a determination process based on theattitude of the terminal device 7. In step S20, the CPU 10 performs aprocess in accordance with the attitude of the terminal device 7 at thesecond timing t2. Specifically, the CPU 10 determines whether theattitude of the terminal device 7 calculated in step S11 is an attitudethat corresponds to the direction determined in step S16 (the directioninstructed by the pirate 92). For example, when it is determined in stepS16 that the arrow 94 is to be shot from the right (that is, the arrow94 is to be shot to the player from the pirate ship 90B), the CPU 10determines whether the attitude of the terminal device 7 is an attitudeas shown in FIG. 20. For example, the CPU 10 determines, based on theattitude data 121, whether a coordinate value for each axis of a unitvector along the Z-axis negative direction of the terminal device 7 iswithin a predetermined range in accordance with the determination instep S16, and thereby determines whether the attitude of the terminaldevice 7 is an attitude in accordance with the instruction by the pirate92.

Further, the CPU 10 determines, in accordance with the determinationresult, an image to be displayed on the LCD 51 of the terminal device 7,and sound to be outputted from the loudspeakers 67 of the terminaldevice 7. Accordingly, for example, when the determination result isaffirmative (when the attitude of the terminal device 7 is the attitudecorresponding to the direction determined in step S16), the processes ofstep S5 and step S7 described below are performed, whereby the image 97and the circular image 98 shown in FIG. 16B are displayed on the LCD 51.Moreover, when the determination result is affirmative, the CPU 10 addspoints.

It should be noted that, in step S20, the CPU 10 determines whether theterminal device 7 is moving, based on the attitude of the terminaldevice 7 calculated using the predetermined number of past frames.Attitudes of the terminal device 7 in the predetermined number of pastframes are stored in the main memory. Therefore, the CPU 10 cancalculate how much the attitude of the terminal device 7 has changed,based on the attitudes of the terminal device 7 in the predeterminednumber of past frames. In the case where the CPU 10 has determined thatthe terminal device 7 is moving, even when the attitude of the terminaldevice 7 is the attitude corresponding to the direction determined instep S16, the CPU 10 generates an image different from that in the casewhere the terminal device 7 is stationary, and displays it on theterminal device 7. FIG. 25 shows an example of an image displayed on theLCD 51 of the terminal device 7 at a second timing t2 when the terminaldevice 7 is moving. As shown in FIG. 25, when the terminal device 7 ismoving, even if the player is holding the terminal device 7 in a properattitude (the attitude in accordance with the instruction by the pirate92), the circular image 98 indicating the position at which the arrowhas been received is displayed at a position shifted from the center ofthe screen. In this case, the player obtains less points than those at atime when the terminal device 7 is determined not to be moving. In thismanner, the game processing is performed in accordance with the attitudeof the terminal device 7 at the second timing t2, and the result of thegame processing differs depending on whether the terminal device 7 isstationary at that timing. It should be noted that, whether the terminaldevice 7 is moving may be determined based on the acceleration detectedby the acceleration sensor 63.

Further, depending on whether the arrow 94 remains without having beenshaken off, the CPU 10 determines, in step S20, positions at which theimage 97 and the circular image 98 are to be displayed. In step S22described below, when a process of shaking off the arrow 94 that hasbeen received with the terminal device 7 is not performed, the arrow 94is not shaken off and remains to be displayed on the LCD 51 of theterminal device 7. Information indicating whether the arrow 94 receivedwith the terminal device 7 is remaining is stored in the main memory.When the arrow 94 is remaining, the image 97 and the circular image 98are not displayed near the center of the screen of the terminal device 7and displayed at positions shifted from the center of the screen asshown in FIG. 25. Also in this case, the user obtains less points thanthose when the arrow 94 is not remaining. Then, the CPU 10 resets theelapsed time which has been measured since step S17, and ends the gamecontrol process shown in FIG. 23.

In step S21, the CPU 10 sets a setting of displaying/not displaying alock-on frame. FIG. 26 is an example of a lock-on frame 99 displayed onthe LCD 51 of the terminal device 7. The process of step S21 isperformed when the arrow 94 has been shot and the current time is not asecond timing t2 (that is, it in a time period from a first timing t1 toa second timing t2). The process of step S21 is a process of determiningwhether the attitude of the terminal device 7 during that time period isa proper attitude (an attitude in accordance with the instruction by thepirate 92) and of displaying the lock-on frame 99 when the attitude isthe proper attitude. The lock-on frame 99 notifies the player that thecurrent attitude of the terminal device 7 is a proper attitude. That is,the CPU 10 determines whether the current attitude of the terminaldevice 7 is the attitude corresponding to the direction instructed bythe pirate 92 when the arrow 94 was shot. When the determination resultis affirmative, the CPU 10 turns on the setting of displaying thelock-on frame 99. When the determination result is negative, the CPU 10turns off the setting of displaying the lock-on frame 99. Afterperforming the process of step S21, the CPU 10 ends the game controlprocess shown in FIG. 23.

On the other hand, in step S22, the CPU 10 performs the process based onaccelerations. In step S22, a predetermined game processing is performedbased on the acceleration detected by the acceleration sensor 63 of theterminal device 7. Specifically, the CPU 10 determines whether the valueof the acceleration of the terminal device 7 in the Z-axis negativedirection is greater than or equal to a predetermined threshold value.When the determination result is affirmative, the CPU 10 performs theprocess of shaking off the arrow 94 received with the terminal device 7,and ends the game control process shown in FIG. 23. Accordingly, theimage 97 and the circular image 98 having been displayed on the LCD 51of the terminal device 7 are not displayed any more. When thedetermination result is negative, or when the arrow 94 has been shakenoff, the CPU 10 directly ends the game control process shown in FIG. 23.

With reference back to FIG. 22, after performing the process of step S3,the CPU 10 performs the process of step S4 next.

In step S4, the CPU 10 performs a process of generating a televisiongame image. Specifically, the CPU 10 obtains a television game image bycausing the first virtual camera to take an image of the virtual space.When an image of the virtual space is taken by the first virtual cameraset at the position and having the angle of view which were set in theprocess of step S12, the pirate 92 is displayed in a zoomed-in manner orzoomed-out manner in accordance with the attitude of the terminal device7. Through these processes, television game images corresponding to thestates of the game, such as a zoomed-in image of the pirate 92 and animage of a scene in which the pirate 92 is shooting the arrow 94, areobtained. Then, the CPU 10 performs the process of step S5.

In step S5, the CPU 10 performs a process of generating a terminal gameimage. Specifically, the CPU 10 generates a terminal game image bycausing the second virtual camera, for which the attitude in the virtualspace has been set in step S14, to take an image of the virtual space.Further, in accordance with the result of the processes of step S20 andstep S22, the CPU 10 superimposes the image 97 and the circular image 98on the generated terminal game image. Moreover, when the setting ofdisplaying the lock-on frame 99 is turned on in step S21, the CPU 10superimposes the image of the lock-on frame 99 on the terminal gameimage obtained by causing the second virtual camera to take an image ofthe virtual space. Thereby, terminal game images corresponding to thestates of the game, such as an image of the virtual space seen from thesecond virtual camera having the attitude in accordance with theattitude of the terminal device 7, an image showing that the arrow 94has hit the screen of the terminal device 7 at the second timing t2 (thecircular image 98, etc.), are obtained. Then, the CPU 10 performs theprocess of step S6.

In step S6, the CPU 10 outputs, to the television 2, the television gameimage generated in step S4. Accordingly, for example, an image such asthat shown in FIG. 13A or FIG. 13B is displayed on the television 2.Moreover, in step S6, sound data is outputted to the television 2, alongwith the television game image, and game sound is outputted at thevolume set in the process of step S13 from the speaker 2 a of thetelevision 2. Specifically, when the attitude of the terminal device 7is the first attitude, the sound volume of the voice of the pirate 92 isincreased than that in the case of the second attitude. That is, whenthe pirate 92 is being displayed in a zoomed-in manner, the sound volumeof the voice of the pirate 92 is increased, and when the pirate 92 isbeing displayed in a zoomed-out manner, the sound volume is decreased.It should be noted that the sound volume of the predetermined music mayvary in accordance with the attitude of the terminal device 7 (in thecase of the first attitude, the sound volume of the predetermined musicmay be increased as in the case of the voice of the pirate 92), oralternatively, may be constant irrespective of the attitude of theterminal device 7. Then, the CPU 10 performs the process of step S7.

In step S7, the CPU 10 transmits the terminal game image to the terminaldevice 7. Specifically, the CPU 10 sends the terminal game imagegenerated in step S5 to the codec LSI 27, and the codec LSI 27 performsa predetermined compression process onto the terminal game image. Thecompressed image data is transmitted to the terminal device 7 via theantenna 29 by the terminal communication module 28. The terminal device7 receives the data of the image transmitted from the game apparatus 3via the wireless module 70. The codec LSI 66 performs a predetermineddecompression process onto the received image data. The decompressedimage data is outputted to the LCD 51. Accordingly, the terminal gameimage is displayed on the LCD 51. Moreover, in step S7, the sound datais transmitted to the terminal device 7, along with the terminal gameimage, and the game sound is outputted form the loudspeakers 67 of theterminal device 7. Then, the CPU 10 performs the process of step S8.

In step S8, the CPU 10 determines whether to end the game. Thedetermination in step S8 is performed, for example, depending on whethera predetermined time period has elapsed from the start of the game, oron whether the user has issued an instruction to end the game. When thedetermination result in step S8 is negative, the process of step S2 isperformed again. On the other hand, when the determination result instep S8 is affirmative, the CPU 10 ends the game processing shown inFIG. 22.

As described above, the player can cause the pirate ship 90A and thelike (including the pirate 92 and the arrow 94) to be displayed in azoomed-in or zoomed-out manner, by changing the attitude of the terminaldevice 7. Specifically, when the player holds the terminal device 7 inan attitude in which the player does not view the LCD 51 of the terminaldevice 7 (the first attitude), the pirate ship 90A and the like aredisplayed in a zoomed-in manner, and concurrently, the volume of thesound outputted form the speaker 2 a of the television 2 is increased.On the other hand, when the attitude of the terminal device 7 is changedinto the second attitude, the pirate ship 90A and the like are displayedin a zoomed-out manner, and concurrently, the volume of the soundoutputted from the speaker 2 a of the television 2 is reduced. Further,the pirate 92 and the arrow 94 are displayed on the television 2, butthe pirate 92 and the arrow 94 are not displayed on the terminal device7. Further, since the terminal device 7 is a portable display device,the dimensions of the screen of the LCD 51 are relatively small, andthus, the pirate ship 90A displayed on the terminal device 7 isdifficult to be viewed, as in the case of the pirate ship 90A displayedin a zoomed-out manner when the attitude of the terminal device 7 is thesecond attitude. Therefore, the player cannot view the instruction fromthe pirate 92 merely by looking at the LCD 51 of the terminal device 7.In other words, the pirate ship 90A displayed on the terminal device 7is always so small that it is difficult to be viewed by the player,whereas the pirate ship 90A displayed on the television 2 becomes easyto be viewed or difficult to be viewed depending on the attitude of theterminal device 7. Therefore, the player needs to look at the television2 in order to view and listen to the instruction from the pirate 92 (theinstruction given in the form of an image and a voice). Further, inorder to easily view and listen to the instructions from the pirate 92,the player views and listens to the television 2 while changing theattitude of the terminal device 7 into the first attitude. After theinstruction has been issued from the pirate 92, the player changes theattitude of the terminal device 7 so as to receive the arrow 94 with theterminal device 7 at the second timing t2. A result indicating whetherthe arrow 94 has been received is displayed on the LCD 51 of theterminal device 7.

As described above, in the game of the exemplary embodiment, it ispossible to cause the player to look at the screen of the television 2and the screen of the terminal device 7 alternately, and to cause theplayer to enjoy the game using the terminal device 7, which is aportable display device.

The television 2 is a stationary display device and the terminal device7 is a portable display device. The portable display device is held bythe player and these two display devices are distanced from each otherto some extent. When performing the rhythm game as described above insuch a game environment, the player can enjoy the rhythm game whichutilizes the distance between these two display devices and thus allowsthe user to feel the broadness of the space.

[7. Modifications]

It should be noted that the above exemplary embodiment is merely anexample, and in another exemplary embodiment, for example, the followingconfiguration may be employed.

In the exemplary embodiment, for example, the pirate ship 90A and thepirate 92 displayed on the television 2 are zoomed in or zoomed outdepending on the attitude of the terminal device 7. Accordingly, theplayer is caused to view and listen to the television 2 and the terminaldevice 7 alternately, to play the game using the two display devices. Inanother exemplary embodiment, for example, fog may be caused toappear/disappear in the virtual space so as to make the pirate ship 90Aand the pirate 92 difficult/easy to be viewed, accordingly. Further, forexample, by blacking out the screen of the television 2, by blurring theentire screen or a predetermined region including the pirate ship 90Aand the like, or by making the pirate ship 90A and the like transparentor translucent, the pirate ship 90A and the like may be made difficultto be viewed (or cannot be viewed). Further, for example, by displayingthe pirate ship 90A and the like in a pixelized manner, the pirate ship90A and the pirate 92 may be made difficult to be viewed (or cannot beviewed). Still further, for example, by displaying another object to thefront of the pirate ship 90A, the pirate ship 90A may be made difficultto be viewed (or cannot be viewed). That is, in the exemplaryembodiment, when the attitude of the terminal device 7 is the firstattitude, the pirate ship 90A and the like are zoomed in so as to bemade easy to be viewed by the player, and when the attitude of theterminal device 7 is the second attitude, the pirate ship 90A and thelike are zoomed out so as to be made difficult to be viewed by theplayer. Thus, by making it difficult for the player to view the pirateship 90A and the like, the exemplary embodiment causes the player tolook at the screen of the terminal device 7. However, in anotherexemplary embodiment, instead of changing the settings of the firstvirtual camera in order to display the pirate ship 90A and the like in azoomed-in/zoomed-out manner, the pirate ship 90A and the like may bemade easy/difficult to be viewed by employing the above describedmethods. In order to make the pirate ship 90A and the like difficult tobe viewed, a process of displaying the pirate ship 90A and the like in azoomed-out manner, and a process of displaying a predetermined image (awhite image representing fog, a black image for blacking out the screen,an image of another object located to the front of the pirate ship 90A,and the like) over a part or the whole of a range including the pirateship 90A and the like may be performed. Further, in order to make thepirate ship 90A and the like difficult to be viewed, a process ofblurring the pirate ship 90A and the like, and a process of displayingthe pirate ship 90A and the like in a pixelized manner may be performed.Accordingly, it is possible to cause the player to look at the screen ofthe television 2 and the screen of the terminal device 7 alternately toplay the game.

Moreover, in the exemplary embodiment, it is determined whether theattitude of the terminal device 7 is the first attitude or the secondattitude. Specifically, it is assumed that the first attitude is anattitude in which the screen of the terminal device 7 is parallel to theground surface (an attitude in which the screen of the terminal device 7is substantially perpendicular to the direction of gravity), and that,in this attitude, the player is looking at the television 2 withoutlooking at the screen of the terminal device 7. Further, it is assumedthat the second attitude is an attitude in which the screen of theterminal device 7 is perpendicular to the ground surface (an attitude inwhich the screen of the terminal device 7 is substantially parallel tothe direction of gravity), and that, in this attitude, the player islooking at the screen of the terminal device 7. That is, in theexemplary embodiment, whether the player is viewing the screen of theterminal device 7 (in other words, whether the player is viewing thetelevision 2) is determined based on the attitude of the terminal device7. In another exemplary embodiment, whether the player is viewing thetelevision 2 (or the terminal device 7) may be determined by anothermethod. For example, an image of the face of the player may be taken bythe camera 56 included in the terminal device 7, and the taken image maybe subjected to a face recognition process. Thus, by determining whetherthe line of sight of the player is directed to the LCD 51 of theterminal device 7, it may be determined whether the player is viewingthe terminal device 7 (whether the player is viewing the television 2).Still further, for example, a camera different from the camera 56 may beprovided in the real space (for example, around the television 2), andthe game apparatus 3 may obtain an image taken by this camera and maydetermine whether the player is viewing the television 2. For example,whether the player is viewing the television 2 can be determined byusing a face recognition technology that determines whether the face ofthe player is included in the image taken by the camera. Still anotherexemplary embodiment, whether the player is viewing the television 2 maybe determined based on whether the player has pressed a predeterminedoperation button of the terminal device 7.

For example, the attitude of the terminal device 7 may be calculated bythe terminal device 7 taking an image of the markers of the markerdevice 6, and based on the calculated attitude of the terminal device 7,whether the player is viewing the terminal device 7 (or the television2) may be determined In this case, a camera for taking an image of themarkers may be provided on the surface opposite to the surface on whichthe LCD 51 of the terminal device is provided. Alternatively, theattitude of the terminal device 7 may be calculated by a camera providedin the real space taking an image of the marker section 55 of theterminal device 7. For example, in the case where a camera provided inthe terminal device 7 takes an image of the two markers of the markerdevice 6, the game apparatus 3 can calculate, based on the positions andthe attitudes of the two markers included in the taken image, whichdirection the terminal device 7 is directed (whether the terminal device7 is facing the television 2), and how much the terminal device 7 isinclined in the lateral direction. Alternatively, in the case where acamera provided in the real space takes an image of the terminal device7, if the terminal device 7 included in the taken image is detected bymeans of image recognition technology such as pattern matching, theattitude of the terminal device 7 can be calculated.

In the exemplary embodiment, the attitude of the terminal device 7 iscalculated based on the angular velocity detected by the gyro sensor,and the attitude of the terminal device 7 is corrected based on theacceleration detected by acceleration sensor. That is, the attitude ofthe terminal device 7 is calculated by using physical amounts detectedby the two types of inertial sensors (the acceleration sensor and thegyro sensor). In another exemplary embodiment, the attitude of theterminal device 7 may be calculated based on the orientation detected bythe magnetic sensor (the direction indicated by the geomagnetismdetected by the magnetic sensor). By use of the magnetic sensor, whichdirection the terminal device 7 is facing (a direction parallel to theground surface) can be detected. In this case, further by use of theacceleration sensor, the inclination relative to the direction ofgravity can be detected, and the attitude of the terminal device 7 inthe three-dimensional space can be calculated.

Further, in another exemplary embodiment, the attitude of the terminaldevice 7 may be calculated based on the physical amounts detected by thegyro sensor 64 and the like in the terminal device 7 and the dataregarding the attitudes may be transmitted to the game apparatus 3.Then, the game apparatus 3 may receive the data from the terminal device7, obtain the attitude of the terminal device 7, and perform the gameprocessing as described above, based on the attitude of the terminaldevice 7. That is, the game apparatus 3 may obtain the attitude of theterminal device 7, by calculating the attitude of the terminal device 7,based on the data corresponding to the physical amounts detected by thegyro sensor 64 and the like from the terminal device 7. Alternatively,the game apparatus 3 may obtain the attitude of the terminal device 7,based on the data regarding the attitude calculated in the terminaldevice 7.

In the exemplary embodiment, when the attitude of the terminal device 7is the first attitude, the pirate ship 90A and the like are displayed ina zoomed-in manner, by moving the position of the first virtual camerain its image taking direction and concurrently reducing the angle ofview of the first virtual camera. In another exemplary embodiment, thepirate ship 90A and the like may be displayed in a zoomed-in/zoomed-outmanner, by changing at least one of the position and the angle of viewof the first virtual. In the exemplary embodiment, when the attitude ofthe terminal device 7 is the second attitude, the position and the angleof view of the first virtual camera are set to substantially the same asthose of the second virtual camera, whereby the imaging ranges of thetwo virtual cameras are made substantially the same with each other. Inanother exemplary embodiment, the position and the angle of view of thefirst virtual camera may not be necessarily substantially the same asthose of the second virtual camera, and as long as the imaging ranges ofthe two virtual cameras are substantially the same with each other, thepositions and the angles of view of the two virtual cameras may beadjusted as appropriate.

In the exemplary embodiment, during a time period from a first timing toa second timing, when the attitude of the terminal device 7 is apredetermined attitude (the attitude in accordance with theinstruction), the lock-on frame 99 is displayed on the terminal device7. In another exemplary embodiment, in addition to the lock-on frame 99(or instead of the lock-on frame 99), a stationary image of the virtualspace taken by the second virtual camera may be displayed on theterminal device 7. For example, when the attitude of the terminal device7 is a predetermined attitude in the above time period, the attitude ofthe second virtual camera may not be changed in accordance with theattitude of the terminal device 7, or the change amount of attitude ofthe second virtual camera may be reduced relative to the change amountof the attitude of the terminal device 7, and then an image of thevirtual space taken by the second virtual camera may be displayed on theterminal device 7. In another exemplary embodiment, in the above timeperiod, the vibrator 79 may be operated in a predetermined pattern inaccordance with a determination result of the attitude. That is, duringthe above time period, whether the attitude of the terminal device 7 isa predetermined attitude may be determined, and a notification inaccordance with the determination result (displaying a frame or astationary image, notifying the user of the determination result bysound, vibration, and the like) may be issued on the terminal device 7.

In the exemplary embodiment, the pirate ship 90A and the like aredisplayed in a zoomed-in or zoomed-out manner in accordance with theattitude of the terminal device 7, and concurrently the volume of thesound outputted from the television 2 is adjusted. Specifically, whenthe attitude of the terminal device 7 is the first attitude (when theattitude in which the player is not viewing the terminal device 7, inother words, the player is viewing the television 2), the pirate ship90A and the like are displayed in a zoomed-in manner and the volume ofthe sound outputted from the television 2 is increased. In anotherexemplary embodiment, when the attitude of the terminal device 7 is thefirst attitude, the volume of the sound outputted from the terminaldevice 7 may be increased or may not be adjusted.

In the exemplary embodiment, the pirate 92 is displayed on thetelevision 2 and the pirate 92 is caused to perform a predeterminedaction at a first timing (an action of shooting the arrow 94, or anaction of pointing a predetermined direction), thereby giving aninstruction to the player. In another exemplary embodiment, theinstruction to be given to the player may take any form, and the objectto be displayed on the television 2 may be any object. In the exemplaryembodiment, it is assumed that the arrow 94 is shot and the shot arrow94 is received with the terminal device 7. However, in another exemplaryembodiment, the game may assume that another object is moved from thetelevision 2 to the terminal device 7, or from the terminal device 7 tothe television 2.

In the exemplary embodiment, an instruction to the user is issued bymeans of an action of the pirate 92 displayed on the television 2 and ofsound outputted from the television 2. However, in another exemplaryembodiment, an instruction by means of either one of an image or soundmay be issued from the television 2.

In the exemplary embodiment, a predetermined instruction is issued tothe player at a first timing, and a game processing is performed basedon the attitude of the terminal device 7 at a second timing which is atiming after a predetermined time period has elapsed from the firsttiming. In another exemplary embodiment, for example, the controller 5may be used as an input device, or the terminal device 7 may be used asan input device. That is, in another exemplary embodiment, the gameprocessing may be performed based on whether a predetermined operationbutton of the controller 5 is being pressed, for example, at the secondtiming, or the game processing may be performed based on whether apredetermined operation button of the terminal device 7 is beingpressed. Moreover, for example, the game processing may be performed inaccordance with the attitude of the controller 5 or the attitude of theterminal device 7, at the second timing.

In another exemplary embodiment, an input by the player may be performedin the form of a gesture (action) of the player himself or herself. Forexample, a camera that takes an image of the player is connected to thegame apparatus 3, and based on the image from the camera, the action ofthe player is determined, whereby an input from the player may beperformed. For example, an instruction is given to the player at a firsttiming, and the player performs an action in accordance with theinstruction at a second timing. Then, based on the image from thecamera, the game apparatus 3 analyzes the action of the player anddetermines whether the action of the player is in accordance with theinstruction.

That is, the game processing may be performed based on the state of aninput to the input device at a second timing. The input state of theinput device may be the state whether a predetermined operation buttonprovided on the controller 5 or the terminal device 7 is being pressed,or may be the attitude of the controller 5 or the terminal device 7itself. The input state of the input device may be a state based on thegesture (action) of the player himself or herself, and the action of theplayer may be determined by the camera taking an image of the player.

Moreover, for example, the game processing may be performed based on asecond timing and a timing at which an input to the input device (forexample, the terminal device 7) is performed. For example, the gameprocessing may be performed based on a difference between a secondtiming and a timing at which a predetermined operation button of theinput device is pressed. Specifically, when the difference is less thanor equal to a predetermined threshold value, a scene in which the arrow94 has reached the terminal device 7 may be displayed on the LCD 51 ofthe terminal device 7, assuming that the player has performed an inputin accordance with the instruction. Moreover, for example, the gameprocessing may be performed based on a difference between a secondtiming and a timing of a predetermined operation performed onto theinput device (an operation that changes the attitude of the input deviceitself, or an operation that accelerates the motion of the input device,such as an operation of shaking the input device). That is, the gameprocessing may be performed based on the difference between the secondtiming and the timing of the input performed onto the input device.

In the exemplary embodiment, a scene in which the arrow 94 is shot fromthe television 2 toward the player is displayed at a first timing, and ascene in which the arrow 94 has reached the terminal device 7 isdisplayed at a second timing. In another exemplary embodiment, a scenein which a predetermined object is moved from the terminal device 7toward the television 2 may be displayed at a first timing, and a scenein which the object reaches the television 2 may be displayed at asecond timing. In this case, the game processing is performed based onthe input state of the input device (the terminal device 7 or anotherinput device) at the second timing. That is, in another exemplaryembodiment, the game as described above may be performed, with thetelevision 2 being a stationary display device switched with theterminal device 7 being a portable display device.

Further, in another exemplary embodiment, a part of the game processingperformed in the game apparatus 3 may be performed in the terminaldevice 7. For example, a virtual space is defined in the terminal device7 and an image of the virtual space is taken by a virtual camera,whereby an image to be displayed on the LCD 51 of the terminal device 7may be generated in the terminal device 7.

Further, in another exemplary embodiment, in a game system including aplurality of information processing apparatuses that can communicatewith each other, the plurality of information processing apparatuses mayshare the game processing performed by the game apparatus 3. Forexample, a game system as described above may be configured by aplurality of information processing apparatuses connected to a networksuch as the Internet. For example, a game system as described above maybe constructed by an information processing apparatus to which theterminal device 7 and a monitor are connected, and a server connected tothe information processing apparatus via the Internet. In this case, theterminal device 7 and the information processing apparatus are arrangedto the player side. Then, for example, operation information based on agame operation performed on the terminal device 7 is transmitted to theserver via the network, and the server performs the game processingbased on the received operation information, and transmits a result ofthe game processing to the information processing apparatus.

Further, in another exemplary embodiment, data may be transmitted andreceived between the game apparatus 3 and the terminal device 7 whichare connected with each other not in a wireless manner but in a wiredmanner. The above described program may be executed in an informationprocessing apparatus, other than the game apparatus 3, for performingvarious information processes, such as a personal computer.

Further, the above game program may not be stored in an optical disc butmay be stored in a storage medium such as a magnetic disc, a nonvolatilememory, and the like. The above game program may be stored in acomputer-readable storage medium such as a RAM or a magnetic disc on aserver connected to a network, and may be provided via the network. Thegame program may be loaded into an information processing apparatus as asource code and may be compiled at the execution of the program.

In the exemplary embodiment, the CPU 10 of the game apparatus 3 executesthe game program, whereby the processes of the flowchart are performed.In another exemplary embodiment, a part or the whole of the processesmay be performed by a dedicated circuit included in the game apparatus3, or by a general-purpose processer. At least one processor may operateas a “programmed logic circuit” for performing the processes.

The systems, devices and apparatuses described herein may include one ormore processors, which may be located in one place or distributed in avariety of places communicating via one or more networks. Suchprocessor(s) can, for example, use conventional 3D graphicstransformations, virtual camera and other techniques to provideappropriate images for display. By way of example and withoutlimitation, the processors can be any of: a processor that is part of oris a separate component co-located with the stationary display and whichcommunicates remotely (e.g., wirelessly) with the movable display; or aprocessor that is part of or is a separate component co-located with themovable display and communicates remotely (e.g., wirelessly) with thestationary display or associated equipment; or a distributed processingarrangement some of which is contained within the movable displayhousing and some of which is co-located with the stationary display, thedistributed portions communicating together via a connection such as awireless or wired network; or a processor(s) located remotely (e.g., inthe cloud) from both the stationary and movable displays andcommunicating with each of them via one or more network connections; orany combination or variation of the above. The processors can beimplemented using one or more general-purpose processors, one or morespecialized graphics processors, or combinations of these. These may besupplemented by specifically-designed ASICs (application specificintegrated circuits) and/or logic circuitry. In the case of adistributed processor architecture or arrangement, appropriate dataexchange and transmission protocols are used to provide low latency andmaintain interactivity, as will be understood by those skilled in theart. Similarly, program instructions, data and other information forimplementing the systems and methods described herein may be stored inone or more on-board and/or removable memory devices. Multiple memorydevices may be part of the same device or different devices, which areco-located or remotely located with respect to each other.

While certain example systems, methods, devices and apparatuses havebeen described herein, it is to be understood that the appended claimsare not to be limited to the systems, methods, devices and apparatusesdisclosed, but on the contrary, are intended to cover variousmodifications and equivalent arrangements included within the spirit andscope of the appended claims.

1. A computer-readable storage medium having stored therein aninformation processing program performed by a computer of an informationprocessing apparatus which causes a stationary display device to displayan image, the program causing the computer to perform: setting a firstvirtual camera in a virtual space in which a predetermined object isarranged; setting a second virtual camera in a virtual space in whichthe predetermined object is arranged; causing the stationary displaydevice to display a first image of the virtual space containing thepredetermined object, the first image being taken by the first virtualcamera; causing a portable display device held by a user to display asecond image of the virtual space taken by the second virtual camera;and determining whether the user is viewing the stationary displaydevice, wherein when it is determined that the user is not viewing thestationary display device, the first image is made more difficult to beviewed than that at a time when it is determined that the user isviewing the stationary display device.
 2. The computer-readable storagemedium having stored therein the information processing programaccording to claim 1, wherein when it is determined that the user is notviewing the stationary display device, the predetermined object in thefirst image is made difficult to be viewed, by performing at least oneof a process of displaying at least the predetermined object in azoomed-out manner, a process of displaying a predetermined image in apart or a whole of a region containing the predetermined object in thefirst image, a process of blurring at least the predetermined object, aprocess of making at least the predetermined object transparent ortranslucent, and a process of displaying at least the predeterminedobject in a pixelized manner.
 3. The computer-readable storage mediumhaving stored therein the information processing program according toclaim 1, wherein the computer is caused to perform obtaining an attitudeof the portable display device, and when the attitude of the portabledisplay device is a first attitude, it is determined that the user isviewing the stationary display device, and when the attitude of theportable display device is a second attitude, it is determined that theuser is not viewing the stationary display device.
 4. Thecomputer-readable storage medium having stored therein the informationprocessing program according to claim 3, wherein when the attitude ofthe portable display device is the second attitude, at least thepredetermined object is displayed in a zoomed-out manner by changing atleast one of a position and an angle of view of the first virtualcamera.
 5. The computer-readable storage medium having stored thereinthe information processing program according to claim 3, wherein anattitude of the second virtual camera is controlled in accordance withthe attitude of the portable display device.
 6. The computer-readablestorage medium having stored therein the information processing programaccording to claim 5, wherein when the attitude of the portable displaydevice is a predetermined reference attitude, an image taking directionof the second virtual camera is set to the same image taking directionas that of the first virtual camera, and the attitude of the secondvirtual camera is changed in accordance with a change of the attitude ofthe portable display device from the reference attitude.
 7. Thecomputer-readable storage medium having stored therein the informationprocessing program according to claim 3, wherein an image takingdirection of the first virtual camera is a predetermined directionindependent of the attitude of the portable display device.
 8. Thecomputer-readable storage medium having stored therein the informationprocessing program according to claim 1, wherein when it is determinedthat the user is not viewing the stationary display device, at least thepredetermined object is displayed in a zoomed-out manner, by changing atleast one of a position and an angle of view of the first virtualcamera.
 9. The computer-readable storage medium having stored thereinthe information processing program according to claim 1, wherein thefirst image contains a second object and the second image does notcontain the second object.
 10. The computer-readable storage mediumhaving stored therein the information processing program according toclaim 1, wherein the predetermined object displayed on the portabledisplay device is more difficult to be viewed than the predeterminedobject that is displayed on the stationary display device when it isdetermined that the user is viewing the stationary display device. 11.The computer-readable storage medium having stored therein theinformation processing program according to claim 1, wherein a positionof the second virtual camera is substantially the same as a position ofthe first virtual camera at a time when it is determined that the useris not viewing the stationary display device.
 12. The computer-readablestorage medium having stored therein the information processing programaccording to claim 1, wherein an angle of view of the second virtualcamera is substantially the same as an angle of view of the firstvirtual camera at a time when it is determined that the user is notviewing the stationary display device.
 13. The computer-readable storagemedium having stored therein the information processing programaccording to claim 1, wherein an imaging range of the second virtualcamera is substantially the same as an imaging range of the firstvirtual camera at a time when it is determined that the user is notviewing the stationary display device.
 14. The computer-readable storagemedium having stored therein the information processing programaccording to claim 3, wherein the second attitude is an attitude inwhich a screen of the portable display device is substantially parallelto the direction of gravity.
 15. The computer-readable storage mediumhaving stored therein the information processing program according toclaim 3, wherein the first attitude is an attitude in which a screen ofthe portable display device is substantially perpendicular to thedirection of gravity.
 16. The computer-readable storage medium havingstored therein the information processing program according to claim 1,wherein the computer is further caused to perform adjusting a volume ofa sound outputted from the stationary display device, in accordance witha result of the determination of whether the user is viewing thestationary display device.
 17. The computer-readable storage mediumhaving stored therein the information processing program according toclaim 16, wherein when it is determined that the user is not viewing thestationary display device, the volume of the sound outputted from thestationary display device is lowered than that at a time when it isdetermined that the user is viewing the stationary display device. 18.The computer-readable storage medium having stored therein theinformation processing program according to claim 3, wherein theportable display device includes at least one of a gyro sensor and anacceleration sensor, and the attitude of the portable display device iscalculated based on data outputted from the at least one of the gyrosensor and the acceleration sensor.
 19. The computer-readable storagemedium having stored therein the information processing programaccording to claim 1, wherein image data indicating the second image isoutputted to the portable display device, and the portable displaydevice includes: an image data obtaining unit that obtains the imagedata outputted from the information processing apparatus; and a displayunit that displays the second image indicated by the image data.
 20. Thecomputer-readable storage medium having stored therein the informationprocessing program according to claim 19, wherein the computer isfurther caused to perform compressing the image data indicating thesecond image, and generating compressed image data, the compressed imagedata is outputted to the portable display device, the image dataobtaining unit obtains the compressed image data outputted from theinformation processing apparatus, the portable display device furtherincludes an image decompression unit that decompresses the compressedimage data, and the display unit displays the second image decompressedby the image decompression unit.
 21. An information processing apparatusthat causes a stationary display device to display an image, theapparatus comprising: a first camera setting unit that sets a firstvirtual camera in a virtual space in which a predetermined object isarranged; a second camera setting unit that sets a second virtual camerain a virtual space in which the predetermined object is arranged; afirst display control unit that causes the stationary display device todisplay a first image of the virtual space containing the predeterminedobject, the first image being taken by the first virtual camera; asecond display control unit that causes a portable display device heldby a user to display a second image of the virtual space taken by thesecond virtual camera; and a determination unit that determines whetherthe user is viewing the stationary display device, wherein when thedetermination unit has determined that the user is not viewing thestationary display device, the first display control unit makes thefirst image more difficult to be viewed than that at a time when thedetermination unit has determined that the user is viewing thestationary display device.
 22. An information processing system thatcauses a stationary display device to display an image, the systemcomprising: a portable display device; a first camera setting unit thatsets a first virtual camera in a virtual space in which a predeterminedobject is arranged; a second camera setting unit that sets a secondvirtual camera in a virtual space in which the predetermined object isarranged; a first display control unit that causes the stationarydisplay device to display a first image of the virtual space containingthe predetermined object, the first image being taken by the firstvirtual camera; a second display control unit that causes the portabledisplay device to display a second image of the virtual space taken bythe second virtual camera; and a determination unit that determineswhether a user is viewing the stationary display device, wherein whenthe determination unit has determined that the user is not viewing thestationary display device, the first display control unit makes thefirst image more difficult to be viewed than that at a time when thedetermination unit has determined that the user is viewing thestationary display device.
 23. An information processing methodperformed in an information processing apparatus that causes astationary display device to display an image, the method comprising thesteps of: setting a first virtual camera in a virtual space in which apredetermined object is arranged; setting a second virtual camera in avirtual space in which the predetermined object is arranged; causing thestationary display device to display a first image of the virtual spacecontaining the predetermined object, the first image being taken by thefirst virtual camera; causing a portable display device held by a userto display a second image of the virtual space taken by the secondvirtual camera; and determining whether the user is viewing thestationary display device, wherein in the step of causing the stationarydisplay device to display the first image, when it is determined thatthe user is not viewing the stationary display device, the first imageis made more difficult to be viewed than that at a time when it isdetermined that the user is viewing the stationary display device.